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ABSTRACT 


The attack on the USS Cole within a civilian port, and the increased threat of 
pirating and terrorism on the high seas, underscore the immediate need for a maritime 
non-lethal capability. This research uses modeling and simulation to explore the 
requirements and tactical use of non-lethal capabilities in a maritime force protection 
mission. Specifically, a multi-agent simulation emulates a tactical-level mission in which 
a U.S. Navy vessel returning to Naval Station, Norfolk, VA, encounters a variety of 
maritime surface threats. Data farming is the method used to address the research 
questions by applying high performance computing to the simulation model, with the 
intent of examining a wide range of possibilities and outcomes. The non-lethal 
capabilities are analyzed in their effectiveness to 1) determine intent, 2) deter inbound 
surface vessels, and 3) engage targets identified as hostile through the continuum of 


force. 
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EXECUTIVE SUMMARY 


The terrorist attack on the USS Cole on October 10, 2000, re-energized national 
efforts towards preserving freedom of the seas and safe access to ports, with a directed 
focus on force protection initiatives and technology. The tremendous potential of non- 
lethal capabilities in maritime force protection has been recognized by the Quadrennial 
Defense Review as well as an independent study conducted by the Naval Studies Board. 
This research, sponsored by the Joint Non-Lethal Weapons Directorate, directly impacts 
the current development of non-lethal requirements and tactics needed for effective 


maritime security. 


To scope the analysis efforts, the following research questions of interest were 


identified for an entering port force protection mission: 


° What non-lethal capabilities are required for a maritime force protection 
mission? 

e When are non-lethal capabilities tactically appropriate? 

e What are the geographical effects? 

e How are non-lethal capabilities used to identify threats from non-threats? 

e Is Multi-Agent Simulation (MAS) an appropriate modeling tool? 


The first step in answering these questions was to develop a scenario within the 
MAS environment that appropriately emulated a Navy ship’s ability to do three tasks: 1) 
Identify potential threats; 2) Determine intent of approaching small vessels; and 3) Deter 


vessels from closing within the 100 yard naval protective zone. 


With the scenario developed, the next step was to apply an efficient design of 
experiment (DOE) using distributed simulation in order to capture a reasonable region of 
possible outcomes. The final scenario was simulated over a range of 33 input variables. 
These input factors included everything from requirements of the non-lethal capabilities 
such as range and firing rates, to quantified intangibles, such as fear and aggression levels 


of inbound targets. 
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Through the power of simulation, over three quarters of a million data points were 


generated. The key element in tying the data to the research questions was the Measures 


of Effectiveness (MOEs) selected for analysis. The following MOEs were defined 


specifically for this research: 


MOE 1 - Deterrence Ratio: The percentage of time the targets are 
deterred. 


MOE 2 — Hostile Identification Ratio: The percentage of time targets are 
identified as hostile, engaged, and subsequently killed, with lethal force. 


MOE 3 — Warning Zone Identification Ratio: If identified as hostile, the 
percentage of time the targets are identified, using non-lethal capabilities, 
outside the threat zone. 


Using multiple data mining techniques, ordinary statistics, and visualization tools, 


the mammoth amount of data was efficiently analyzed to provide insight on the research 


questions. Key insights into non-lethal requirements and tactical application include: 


The employment of non-lethal capabilities is extremely effective when 
used to identify threats from non-threats in an ambiguous situation. 


Inbound speed is the critical factor in identifying and engaging inbound 
hostile threats outside of the exclusion zone. 


The number of inbound targets has little to no impact on identification and 
engagement rates of hostile targets. 


The first response non-lethal capability is the most crucial in deterring 
non-suicidal targets. 


The AHD is significantly more effective when employment time is less 
than 30 seconds against hostile targets and 1 minute for neutral or loitering 
targets. 


When used alone, counter-personnel non-lethal capabilities fail to deter 
loitering targets who attack when within close proximity. 


With a limited number of non-lethal capabilities applied to a very specific 


mission, this thesis researched what other possible applications are appropriate using the 


methodology applied. Follow-on work was identified for three primary research areas: 


Requirements — This research modeled three counter-personnel non-lethal 


capabilities. Future work should include scenario expansion to include counter-material 


capabilities. 
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Tactics — The scenario simulated involved one primary tactic adapted from 
current U.S. Navy tactics. Future work should include a comparative analysis of new 


tactics, especially in areas where this research deemed current tactics fall short. 


Vulnerability Assessments — In addition to exploring the requirements and tactics 
of the non-lethal capabilities, this research was very effective in exploring the geographic 
vulnerabilities of the modeled port. Given this success, future work should apply this 
methodology to other geographic ports or chokepoints of interest in order to assist in anti- 


terrorism/force protection planning prior to ship arrival. 


In summary, this research used cutting edge modeling and simulation to 
effectively emulate a complex scenario where little historical performance data exists. 
This thesis produced valuable insights through application of proven operations research 
tools and techniques, and provides a revolutionary compliment to subject matter expertise 


in non-lethal requirements and tactics development early in the acquisition process. 
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I. INTRODUCTION 


Preserving the freedom of the seas is a top national priority. The right of 
vessels to travel freely in international waters, engage in innocent and 
transit passage, and have access to ports is an essential element of national 
security. * 


The National Strategy for Maritime Security 


A. BACKGROUND OF NON-LETHAL CAPABILITIES 

i ke Defining the Need for Non-Lethal Capabilities 

Libya, Lebanon, Panama, Somalia, Bosnia, Haiti, and Kosovo: Over the past 20 
years, these are the countries that have been the theater of operations for the U.S. 
military. Yet, despite the presence and action of the U.S. military, none of these 
operations were deemed wars. In fact, the last Congressional declaration of war was 
World War II (Declaration of War by the United States, 2006), yet U.S. military power 
has been projected many times since. After the Cold War, the U.S. shifted its policies 
when committing its forces. Specifically, U.S. decision makers expanded from the use of 
military force strictly for conventional war to the application of forces over a wide 
spectrum of military operations. This shift in the application of military force has had a 


significant impact on training, doctrine, and capabilities of U.S. forces. 


This is most clearly exemplified in the differing national strategies between the 
two eras. NSC-68, which was the guiding principal for national strategic objectives in 
the Cold War, defines four possible courses of action: 1) Continuation of current 
policies; 2) Isolation; 3) War; and 4) A more rapid building up of the political, economic, 
and military strength (National Security Council, 1950). Of these, war is the only action 


which incorporates actual use of military forces. 


In reaction to the first attack on the World Trade Center, the two attacks on U.S. 
embassies in Africa, the assault on the USS Cole, and finally, the mass terrorist attack on 


the Twin Towers and the Pentagon in September 2001, the administration adopted a new 


* Department of Defense and Homeland Security. (2005). The National Strategy for Maritime Security. 
Washington D.C. 
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approach towards the National Security Strategy that fundamentally differed from any 
seen before. Specifically, the Bush administration embraced preventive war. The most 
significant point of this National Security Strategy is the motivation to go beyond 
preemptive strikes and adopt a policy towards preventative action before threats even 


have the chance to develop. 


Given the U.S. military’s experience in non-wartime situations, military doctrine 
encompasses the concepts of the current National Security Strategy. The range of 
military operations (ROMO), as shown in Figure 1, describes military operations 


extending from war to military operations other than war (MOOTW). 


RANGE OF MILITARY OPERATIONS 
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Figure 1. Range of Military Operations (From Chairman Joint Chief of Staff, 1995) 


The idea of preventative action is captured within the purpose of MOOTW, which 
is to deter war and promote peace. Because MOOTW are preventative in nature and 
generally involve non-combatant civilians, they follow more restrictive rules of 
engagement (ROE) (Chairman Joint Chief of Staff, 1995). As such, non-lethal 
capabilities have been identified as a significant tool in accomplishing MOOTW 


missions, as well as enhancing mission effectiveness in conventional combat operations. 


2 History of Non-Lethal Capabilities Development 

Although numerous examples of non-lethal employment can be extrapolated from 
history, the first formal instance of non-lethal capabilities directly utilized to support a 
U.S. military objective is the U.S. intervention in Somalia. LtGen Anthony Zinni, tasked 
with the withdrawal of UN troops from Mogadishu, called for the immediate fielding of 
non-lethal capabilities in order to minimize casualties and mitigate escalation (Non-lethal 


Weapons, 2005). 


By 1996, the Department of Defense released Directive 3000.3, officially 
establishing the Joint Non-Lethal Weapons Program (JNLWP), Non-Lethal policies, and 
DOD-wide responsibilities for non-lethal development. From this, the Commandant of 
the Marine Corps, serving as the executive agent (EA) for JNLWP, established the Joint 
Non-Lethal Weapons Directorate (JNLWD). The JNLWD serves as the central liaison 
for all issues involving non-lethal capabilities, including research and development, 
science and technology, requirements, and acquisition of non-lethal technologies (The 


Joint Non-Lethal Weapons Program, 2005). 


In December 2002, the Defense Department’s Joint Requirements Oversight 
Council (JROC) approved a Mission Need Statement (MNS) expressing the need for non- 
lethal capabilities in order to “break the cycle of violence that often prolongs or escalates 
conflict” (Allison et al., 2004). In 2004, the Council on Foreign Relations released a 
report recommending more intensive use of current non-lethal capabilities as well as 
aggressive development of new non-lethal concepts (Allison et al., 2004). In response, 
the JNLWP EA issued a directive requiring a Capabilities-Based Assessment (CBA) for 
all non-lethal weapons and capabilities. In accordance with the Chairman of the Joint 
Chiefs of Staff Instruction 3170.01 for the Joint Capabilities Integration and 
Development System (JCIDS), the JNLWD spearheaded a JCIDS analysis. The 
objectives of this analysis were to meet the EA’s directive and update the previous JROC 
MNS (Functional Area Analysis, 2004). Figure 2 depicts the objective of the JCIDS 


analysis at each step. 





Figure 2. The Four Steps in the JCIDS Process (From Functional Area Analysis, 2004) 


3 Non-Lethal Capabilities and the Maritime Environment 

The attack on the USS Cole within a civilian port, and the increased threat of 
pirating and terrorism on the high seas, underscored the immediate need for a maritime 
non-lethal capability. This research seeks to focus on non-lethal capabilities in the 
maritime environment, and therefore will only address elements of the JCIDS analysis 
which focus on the key maritime terrain. The JCIDS’ Functional Area Analysis (FAA) 
determined which maritime tasks can be supported by non-lethal capabilities. The 
conclusion was the identification of 18 tasks that apply to the maritime terrain. The FAA 
validated the relevance of non-lethal capabilities in accomplishing critical missions at 
sea. The second JCIDS step was to conduct the Functional Needs Analysis (FNA), one 
of the two major findings identified that current and projected non-lethal capabilities fail 
to satisfy conditions involving maritime tasks. Further conclusions showed that 
performance gaps existed in most standards required of non-lethal capabilities to include 
range, coverage, duration, effects and reversibility (Functional Needs Analysis, 2004). 
Based on the gaps identified in the FNA, the Functional Solutions Analysis (FSA) 
focused on material and non-material solutions needed for mitigation. The primary 
finding of the FSA was, “Gaps identified during the FNA are mainly caused by the 
physical limitations of current non-lethal capabilities” (Functional Solutions Analysis, 


2005). 


In a separate assessment, the Naval Studies Board defined the need for non-lethal 
capabilities in a variety of U.S. Navy missions. The assessment divided the missions into 
defensive and offensive objectives. The primary defensive mission identified was force 
protection while the offensive missions included maritime interdiction, blockades and 
strikes (Naval Studies Board, 2003). In its conclusions, the assessment noted four areas 
of concern. The following two contribute to the underlying motivation for this research: 


e Without a much stronger overall program to understand and characterize 
the effects and effectiveness of non-lethal weapons, commanders will 
remain reluctant to request or employ them (Naval Studies Board, 2003). 


e Without concepts for the use of non-lethal weapons, developers will not be 
successful in focusing ideas and programs (Naval Studies Board, 2003). 


B. OBJECTIVE 

The JCIDS FSA conclusion that new non-lethal capabilities are needed and the 
Naval Studies Board assessment that understanding effectiveness and developed concepts 
are critical for non-lethal implementation drive the main questions investigated in this 
work. This research seeks to use modeling and simulation to explore the requirements, 
effectiveness and tactical use of non-lethal capabilities for a maritime force protection 
mission. 
C. SCOPE OF THESIS AND RESEARCH QUESTIONS 

This research will focus on a tactical-level scenario in which a U.S. Navy vessel 
returning to Naval Station (NAVSTA), Norfolk, VA, will encounter a variety of maritime 
surface threats. The objective of the U.S. Navy vessel is to reach the terminal point 
(NAVSTA pier). En route to the objective point, the U.S. Navy vessel will employ non- 


lethal capabilities as appropriate. 


The primary research question of this thesis is: 


e What non-lethal capabilities are required in a maritime force protection 
environment in order to effectively determine intent and/or deter 
suspicious small vessels? 


Supporting questions are as follows: 


e Is a multi-agent simulation (MAS) an appropriate means to represent and 
analyze a ship’s ability to determine intent and use continuum of force? 


e In what region of conditions are non-lethal capabilities tactically 
appropriate against high speed hostile threats? 
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e What, if any, are the major differences in requirements given varying 
geographic constraints? 


e What requirements are needed of non-lethal capabilities in order to 
effectively separate threats from non-threats in a homogeneous group? 


D. METHODOLOGY 

The systematic approach of this research is to apply a process called data farming, 
an analytic process initially designed to support research and development efforts of the 
Marine Corps Warfighting Laboratory’s Project Albert. Data farming is a method used to 
address decision makers’ questions by applying high performance computing to a 
simulation model, with the intent of examining a wide range of possibilities and 
outcomes (Project Albert, 2006). By incorporating cutting edge experimental designs 
developed at the Naval Postgraduate School into the data farming process, thousands of 
data points are collected and analyzed in order to identify significant factors, interactions 
between key variables, and critical performance thresholds (Cioppa, 2002; Kleijnen et al., 


2005; Cioppa and Lucas, 2006). 


The following chapters provide the specific background on each facet of the data 
farming process as applied to non-lethal capabilities in the maritime environment. 
Chapter II describes the tactical scenario designed for this research. The chapter 
discusses the assumptions and methodology used to emulate tactically-significant 
interactions between maritime vessels using a multi-agent simulation (MAS). Chapter III 
addresses how a robust set of data is created by implementing an efficient design of 
experiment (DOE) and leveraging high power computing power. Details include which 
initial conditions and parameters will be varied in order to explore the wide spectrum of 
possible real-world outcomes. Chapter IV focuses on the analysis of the data after the 
DOE is applied. Specifically, this chapter discusses the motivation behind the measures 
of effectiveness (MOE) selected, how these MOEs are extracted from the simulation 
process, and what statistical insight is garnered in answering the primary research and 


future applications of this research. 


Il. SCENARIO DESCRIPTION 


It is impossible to foresee, or to fully comprehend, all the challenges we 
will face. But by building a balanced force that is resilient and adaptable, 
with the depth of capabilities required to meet the demands of a multi- 
mission, multi-task environment, we can mitigate against this uncertainty.’ 


Admiral Mike Mullen 


A. OVERVIEW 

The primary objective of this research is to use modeling and simulation to 
capture the required non-lethal capabilities to deter terrorist attack or determine hostile 
intent of maritime assets. A realistic scenario within the modeling and simulation 
environment provides the foundation for this analysis. This chapter describes the model 
selection and scenario design implemented for this specific research. 
B. MODEL SELECTION 

1, Multiple Agent Simulations 

Determination of intent and deterrence of a target are both decisions made by one 
side or the other. In the real world, many factors such as weather, terrain, and the 
physical capabilities and limitations of the players drive the decision making process. 
These features can be captured in a physics-based or probability-based model. However, 
human factors, such as morale, confidence, fatigue, and intelligence, are also critical 
elements of the hostile decision to continue the attack or be deterred. They are also key 
factors in the shipboard decision to recognize the threat and take non-lethal or lethal 
action. The previously mentioned models generally ignore these factors. For this reason, 
this research uses multi-agent simulation (MAS) in order to capture the intangibles, such 
as fear and aggression, which are essential to effectively emulate the decision making 


processes. 


Although definitions vary, the primary distinction of MAS is that it is made up of 
agents, or entities, that behave autonomously and are capable of choice given a simple set 
of rules or logic (Sanchez and Lucas, 2002). Like probability-based models, MAS 
incorporate probabilities in order to account for random effects. One goal for this 


T Mike G. Mullen, (2006). CNO Guidance for 2006: Meeting the Challenge for a New Era. Retrieved 
July 8, 2006 from http://www. globalsecurity.org/military/library/policy/navy/cno-guidance_2006.html. 
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scenario is to simulate the choice of a surface target to deter. This choice is based on 
how influenced the target is by, or how ‘afraid’ it is of, the U.S. Navy ship. A target’s 
decision whether or not to deter is also a factor of how aggressive that target is. For 
example, a target that is hostile in nature will show more aggressiveness and less fear 
than a neutral target, and consequently will be less likely to deter. These levels of fear 
and aggression which drive agents’ choices and resultant actions are captured using 
MAS. 

2. Why PYTHAGORAS? 

The MAS chosen for this research is PYTHAGORAS, a Northrop Grumman 
product. Three reasons influence the selection of this model. First, PYTHAGORAS 
offers a unique set of capabilities appropriate for this research (Bitinas et al., 2006). 


Specifically, PYTHAGORAS 


e Incorporates soft rules to distinguish unique agents; 
e Uses desires to motivate agents into moving and shooting; 
e Includes the concept of affiliation (established by sidedness, or color 


value) to differentiate agents into members of a unit, friendly agents, 
neutrals, or enemies; 


e Allows for behavior-changing events and actions (called triggers) that 
may be invoked in response to simulation activities; and 


e Retains traditional weapons, sensors, and terrain. 
Second, PYTHAGORAS is specifically designed to be compatible with data 
farming, the primary method used for data collection and analysis in this research (see 


Chapter IV). 


Third, as a U.S. developed model, this research can be applied to quickly assist 
our operating forces in exploiting vulnerabilities and determining tactics to mitigate risk 
within ports and choke points throughout the world. 

C; MODEL CONFIGURATION 

1. Scale 

The first step in building a scenario in PYTHAGORAS is defining what a time 
step and pixel distance mean. All subsequent terrain features, weapons and sensor 
ranges, firing rates, and speeds depend on the pixel distance and/or the amount of time 


encompassed by a time step (Bitinas et al., 2006). 
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The original intent was to create a scenario representing the entrance to Thimble 
Shoals Channel, Norfolk, VA, to the Elizabeth River, Norfolk, VA. This area covers a 25 
nautical mile by 15 nautical mile area. When translated to the 1000 pixel by 1000 pixel 
PYTHAGORAS terrain box, each pixel represented nearly 50 yards. In a real-world 
scenario involving a high speed small boat, much can occur over 50 yards. In order to 


capture more range fidelity, the real-world area of interest was refined. 


The scenario was re-scaled to a 9.5 nautical mile by 9.5 nautical mile area, 
focusing on the entrance to the Elizabeth River from Thimble Shoals Channel. 
Translating onto the 1000 pixel by 1000 pixel terrain box, each pixel represents 
approximately 19 yards. This offers almost three times as much range fidelity. With a 
time step defined as 0.25 minutes, the blue ship takes approximately 85 minutes (300 
time steps) to reach its objective point. Table 1 lists how various units of distance, time 
and speed are represented by the defined pixel and time step measurements. 

1.10 


Distaneé 19.36 | 193.60 | 1936.00 
Ki 1.77 





1770.28 
00 
Time Minutes 0.25 1.00 100.00 
Hours | 0.00 | 0.02, [1.67 
Speed 


Table 1. Pixel and Time Step Translations 











2. Terrain 

PYTHAGORAS terrain features offer many options to include height, protection, 
concealment, and trafficability. For this scenario, grey terrain represents land. No agents 
can move through land and neither detections nor engagements can occur through land. 
Yellow terrain represents land features which an agent cannot pass through, but do not 
offer 100% concealment as grey terrain does. Blue terrain represents water. It is 


assumed that the blue ship will remain on its track within the channel and all small boat 
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threats have sufficiently small drafts to traverse across water of any depth. Therefore, the 
blue terrain is 100% trafficable and all targets are detectable when within sensor range. 
Figure 3 compares the actual area of interest versus the terrain constructed in the 


PYTHAGORAS model. 





Figure 3. Comparison of Actual Terrain (From Graphic Retrieved from Google Maps, 
2006) and PYTHAGORAS Model Terrain 

D. BLUE AGENTS 

In PYTHAGORAS, an agent class refers to a group of agents that share the same 
desires and capabilities. Although PYTHAGORAS allows for complex maneuvering, all 
blue (U.S.) agents are modeled to move together, consistent with current naval policies 
and procedures, at a constant speed along a point-to-point Plan of Intended Movement 
(PIM). The purpose for this maneuvering scheme is twofold. First, although observing a 
target’s reaction to own-ship maneuvers is an effective means in determining intent, it is 
not always possible. In an entering port scenario, it is realistic that the entering ship is in 
a restricted channel operating under speed restrictions. In this situation, a U.S. Navy 
vessel must rely on non-lethal capabilities and communications to determine intent. The 
second reason is to prevent so much complexity in the model that the effects of the non- 
lethal capabilities cannot be analyzed effectively. Figure 4 highlights the PIM of the blue 


ship throughout the scenario. 
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Figure 4. Blue Ship PIM 


For this scenario, all blue agents represent a single U.S. Navy ship. Each blue 
agent represents a capability, either non-lethal or lethal. The motivation for creating 
separate agent classes for each capability is to capture the rules of engagement (ROE) in 
the scenario. In other words, by defining different agent classes, the blue ship can assign 
the appropriate capability to different types of targets. For example, a target that is 
recently detected and has not been queried is eligible to be engaged by the Acoustic 
Hailing Device, but not by a lethal weapon. The five blue agent classes are: 1) Acoustic 
Hailing Device (AHD), 2) Optical Dazzler (OD), 3) Warning Munitions (WM), 4) Lethal, 
and 5) Lethal Exclusion Zone (EZ). 

iF Non-Lethal Capabilities 

a. Acoustic Hailing Device 

AHDs use directed acoustic energy to provide a non-lethal warning 
capability by producing highly directional sound beams. These sound beams can be 
formed into either warning tones or voice commands (AHD Fact Sheet, 2006). In 
November, 2005, the cruise ship “Seaborne Spirit” used an AHD while successfully 
repelling armed pirates off the coast of Somalia (Long Range Acoustic Device, 2006). 
This operational use in the maritime environment inspires further research on the 
effectiveness of this capability in the key terrain of sea. It is modeled in this scenario as a 
direct capability with varying firing rates (depending on if its use is for a warning tone or 
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voice commands). Figure 5 shows the Long Range Acoustic Device (LRAD), an AHD 
developed by American Technology Corporation, in use onboard a U.S. Navy patrol 


craft. 





Figure 5. |The LRAD onboard USS Typhoon (From Long Range Acoustic Device, 
2006) 

b. Optical Dazzler 

An OD is a directed-energy capability that employs intense visible light, 
usually generated by a laser (Dazzler, 2006). The purpose of an optical dazzler is to 
provide visual warning and, as a target closes, optical disorientation. Where the AHD 
provides the blue ship with an audio capability, the OD is included to provide the blue 
ship with a visual capability. Since a dazzler are typically steady or strobe light, it is 
modeled in this scenario as a direct capability with a high rate of fire. 

c: Warning Munitions 

WWM’s are projectile cartridges that create a bright flash, loud bang, and 
smoke when fired down range (JNLWM Fact Sheet, 2006). Although audio and visual 
capabilities are already assigned to the blue ship, the WMs model a more aggressive 
warning posture. They are modeled in this scenario as an indirect capability where a 
circular error probability (CEP) radius originates from the aim point. All targets within 
the CEP radius are affected by the WMs accordingly. 

2s Lethal Capabilities 

a. Lethal 

Because this research focuses on the determination of intent and the 
escalation of force up to lethal means, the scenario does not attempt to model the lethality 
of a specific weapon type. Once a threat has been identified as a hostile threat, it is 


engaged with a ‘one shot, one kill’ lethal weapon. Although realistically a U.S. Navy 
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ship does not shoot and kill every target identified as hostile, this feature of the model is 
used later for data analysis. See Chapter 4 for a more detailed discussion. 
b. Lethal Exclusion Zone 
Navy tactics dictate that a target can be assumed to be hostile if it ignores 

all warnings and continues to close within a certain distance. To capture this within the 
model, a second lethal weapon is defined. The lethal (EZ) weapon is fired when any 
contact closes within 100 yards of the blue ship. This distance derives from the U.S. 
Coast Guard regulation on naval vessel protective zones stating that no vessel is allowed 
within 100 yards of a U.S. naval vessel without authorization (Naval Protective Zones, 
2006). The purpose in having two lethal weapons defined is to extract which contacts are 
deemed hostile due to proximity versus those identified as hostile independent of range. 
E. MODELING NON-LETHAL CAPABILITIES 

iL Non-Lethal Capabilities Employment Tactics 

The tactics used by the blue ship to engage contacts with non-lethal capabilities 
reflect the current U.S. Navy force protection architecture. Specifically, three zones are 
created around the blue ship for a layered defense (Naval Studies Board, 2003). In the 
first zone, the assessment zone, the blue ship must detect and assess if a target is a 
potential threat. At scenario start, the blue ship identifies all targets as neutral. The AHD 
is used to make an initial query and warn targets. In the second zone, the warning zone, 
the blue ship uses some or all of its non-lethal capabilities to deter continuing threats. Ifa 
target reaches the third zone, the threat zone, the intent of the approaching craft is 
assumed to be hostile (Naval Studies Board, 2003) and it is engaged with the lethal (EZ) 


weapon. Figure 6 illustrates the force protection architecture for this scenario. 
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Figure 6. | Layered Defense against Small Boats (After Chief of Naval Operations, 
2003) 


To quote the Department of Defense (DOD) Directive 3000.3 “Policy for Non- 
Lethal Weapons,” released on July 9, 1996, and recertified on November 21, 2003: 


Neither the presence nor the potential effect of non-lethal weapons shall 
constitute an obligation for their employment or a higher standard for 
employment of force than provided for by applicable law. In all cases, the 
United States retains the option for immediate use of lethal weapons, when 
appropriate, consistent with international law (Department of Defense, 
2003). 


The scenario captures this directive with the use of the lethal weapon. Regardless 
of range or what non-lethal capabilities have or have not been used, the blue ship will 
engage with lethal force in the first or second zone once the determination of hostile 
intent has been made. Appendix A demonstrates in detail how the PYTHAGORAS 
sidedness properties are used to model the continuum of force. 

2. Non-Lethal Capabilities Effects 

When talking about effects, this scenario does not try to capture the physical 
reaction of the non-lethal capabilities on a person or piece of equipment. The effects in 
this scenario refer to the targeting and maneuvering decisions of the agents in reaction to 
the employment of non-lethal capabilities. The following paragraphs describe the overall 
concept of these effects and how they control the blue ship’s determination of hostile 


intent and a target’s decision to deter. 
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a. Suspicion Level 

The suspicion level represents how threatening a target is to the blue ship. 
Each time a non-lethal capability is employed against a target, and that target does not 
react or deter, the suspicion level against that target is raised. 

b. Suspicion Level Threshold 

The suspicion level threshold is the amount of suspicion needed against a 
target for the blue ship to take further action. The most significant threshold is when 
enough suspicion has been built to declare hostile intent and escalate to lethal force. Sub- 
levels represent the continuum of force where the blue ship is triggered to use more 
aggressive capabilities. 

c. Deterrence Level 

The deterrence level represents how influenced a target is when engaged 
by non-lethal capabilities. This level is cumulatively collected for every type and 
instance of non-lethal capability employment against a target. 

d. Deterrence Level Threshold 

The deterrence level threshold is the amount of cumulative influence 
needed for a target to deter. For all threat types, once deterred, a target will maneuver 
away from the blue ship, regardless of initial movement desires. 

3. Hierarchy of Threats 
In the real world, a U.S. Navy ship is not equally suspicious of all inbound 

targets. Watchstanders use the kinematics of a particular threat in determining its 
suspicion level. Two important kinematics used to determine intent are the number of 
targets grouped together and the speed of those targets. To capture this within the 
scenario, the blue ship groups inbound threats into one of four categories. Category I 
threats incite higher levels of suspicion than Category IV threats when blue ship warning 
and deterrence efforts are unheeded. Figure 7 defines the criteria for each threat category 


based on target numbers and speeds. 
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* Category I: 3 or more targets AND 
faster than 18 knots 


* Category II: 3 or more targets OR 
faster than 18 knots 


* Category III: 2 or less targets AND 
slower than 18 knots 


* Category IV: Fishing or loitering 
targets 





Figure 7. Hierarchy of Threats 


F. THREAT AGENTS 

The purpose of the threat agents in this scenario is to provide the blue ship with a 
realistic selection of threats that a U.S. Navy vessel could face when entering a port. It is 
important to understand that the naming convention used for these threats (hostile, 
neutral, and suspect) refers to the target’s true identity (ground truth) and is completely 
independent of the blue ship’s classification of the target. 

1. Hostile Targets 

Hostile targets are designed to maneuver on a direct attack run towards the blue 
ship. In other words, they act as “suicide boats.” As such, they have a higher deterrence 
threshold, meaning that they are less likely to be deterred by non-lethal capabilities. 
However, once the deterrence threshold is reached, hostile targets will break off the 
attack run for a variable amount of time, and then recommence the attack run. When a 
hostile target re-commits to an attack run, the blue ship retains all prior suspicion levels 
built up against that target. Conversely, the hostile target’s deterrence level is re- 


initialized below the deterrence level threshold. 


There are two groups of hostile targets in the scenario which represent two 
different attack tactics. The first group conducts an open water approach, giving the blue 
force early target detection and a longer prosecution time. The second group conducts a 


more covert maneuvering scheme. Here the enemy hides behind a jetty, denying early 
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detection. When the blue ship nears its closest point of approach, the second group 
abandons its hiding position and commences its attack run. Figure 8 illustrates both 


hostile maneuvering tactics. 





Figure 8. __ Hostile Targets’ Movement 


2. Neutral Targets 

Neutral targets, like the blue ship, have their own PIM. However, since the 
neutral targets are closing in on the blue ship when detected and will pass within close 
proximity, the blue ship must decide if the approaching ship has ill intent or is simply a 
passerby. Neutral boats have a lower deterrence threshold and are more likely to be 
deterred by non-lethal capabilities. After the deterrence threshold is reached, a neutral 
target will still proceed to its next waypoint, but will maneuver as necessary to avoid the 


blue ship. 


There are two types of neutral targets in the scenario which represent two 
different geographic situations when the blue ship reacts to the closing target. The first 
type represents a channel approach. Although the blue ship has early target detection, 
this situation forces the blue ship to deal with multiple types of threats at once when 
transiting through the chokepoint. The second type of neutral target represents a crossing 


ferry. Although neutral targets are more easily deterred, this situation poses a geographic 


17 


constraint where the neutral target may not have room to clearly maneuver away from the 
blue ship even though its deterrence threshold is met. Figure 9 illustrates both neutral 


maneuvering schemes. 





Figure 9. Neutral Targets’ Movement 


3. Neutral Fishing/Loitering Targets 

These targets, like those described in the previous section, have a lower 
deterrence threshold. Yet instead of following a PIM, the neutral fishers loiter around a 
chokepoint through which the blue ship must transit. Additionally, they respond as a 
group in that once one of the targets reaches its deterrence threshold, the entire group will 
move to avoid the blue ship. 

4. Suspect Fishing/Loitering Targets 

The suspect fishers represent a class of targets that are not as aggressive or overt 
as the hostile targets, but still desire to attack the blue ship. At scenario start, they loiter 
with the neutral fishers at a chokepoint and await a “target of opportunity.” This creates 
the appearance of a homogenous group that is actually composed of different classes of 
threats. Once the blue ship moves within a close range, a suspect fisher will change its 
scheme of maneuver and commence a direct attack run on the blue ship. Unlike the 


hostile targets, once a suspect fisher reaches its deterrence threshold, it will move as 
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necessary to avoid the blue ship (and once it is deterred, it will remain deterred). Figure 
10 below highlights the chokepoint area where both neutral and suspect fishers are 


positioned. 


> |> ut a (2 aot Tt 








Figure 10. Chokepoint with Loitering Targets 


G. SUMMARY 

Although this scenario represents a range of threats, a single run represents only 
one possible outcome. Since little or no data exists on the reaction of surface maritime 
threats to newly developed non-lethal capabilities and accurately predicting the human 
factor inputs is problematic, a single run is not sufficient for this research. The next 
chapter discusses how an efficient design of experiment (DOE) and high power 
computing capabilities are used to analyze the trends in output given a wide variety of 


possible reactions from a target. 
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Hl. ANALYSIS METHODOLOGY 


Any intelligent fool can make things bigger, more complex, and more 
violent. It takes a touch of genius, and a lot of courage, to move in the 
opposite direction.’ 


Albert Einstein 


A. OVERVIEW 

This thesis does not explore the physiological effects of non-lethal capabilities on 
personnel. The ‘effectiveness’ in this scenario relates to the impact non-lethal 
capabilities have on the tactical objectives to 1) determine intent, 2) deter inbound surface 


vessels, and 3) engage targets identified as hostile with lethal force. 


Figure 11 represents modeling and simulation as a function where the output of 
the simulation measures the effects of the inputs. Within the simulation environment, a 
set of input parameters are applied to the model and the output is recorded. The input 
parameters are then varied, and the model is run again. This iterative process is repeated 
until every input set of interest is applied to the model. The resultant Y vector is 
composed of literally thousands of values, representing the associated output for each set 
of input parameters. The methodology of this thesis is to build the right-side of the 
equation so that the resultant output is pertinent in answering the initial research 


questions. 


a i a 
Y= FOO he miter 


Output Input & Model 
Parameters 


Figure 11. A Model/Simulation as a Big Function (From OA4655, 2005) 


B. MEASURES OF EFFECTIVENESS 
To accomplish this, the first step is to determine what output is needed for useful 
analysis. One type of output is a measure of effectiveness (MOE). A MOE defined is a 


quantitative measure, generated by the model, used to compare the effectiveness of 


+Albert Einstein: Albert Einstein Quotes. Retrieved July 21, 2006 from 
http://www.quotationspage.com/quote/24921.html. 
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alternatives in achieving the tactical objectives (OA4655, 2005). The following lists the 


MOEs designated for this research and to be applied in follow-on analysis. 


e MOE 1 - Deterrence Ratio: The percentage of time the targets are 
deterred. 
e MOE 2 — Hostile Identification Ratio: The percentage of time targets are 


identified as hostile, engaged, and subsequently killed, with lethal force. 
This MOE not only captures accurate identification of hostile threats, but 
misidentification of neutral targets. 


° MOE 3 — Warning Zone Identification Ratio: If identified as hostile, the 
percentage of time the targets are identified, using non-lethal capabilities, 
outside the threat zone. 


C; VARIABLES OF INTEREST 

With the PYTHAGORAS model selected and the scenario constructed, the next 
step is to define which input variables, or factors, are of interest. Table 2 outlines the 
factors chosen for the experimental design. Each factor is varied over a range in order to 
evaluate the underlying effects on the output. Each factor type is described in detail 


below. 


Ze 


Factor # Factor ID Factor Name Pythagoras 
Value 


AHDMaxRange AHD Maximum Range 25 (484 yds) 80 (1580 yds) 
ODMaxRange OD Maximum Range 15 (290 yds) 55 (1065 yds) 
WMMaxRange WM Maximum Range 5 (96 yds) 25 (484 yds) 





AHDFirRate AHD Firing Rate 0.17 (1.5 min) 0.5 (0.5 min) 
ODRedRadRat OD Red Radius Ratio 
WMRedRadRat WM Red Radius Ratio 
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I, Blue Factors 

a. AHD Maximum Range 

This is the maximum range of the AHD. Although different AHDs exist 
and their capabilities are known, looking at variable maximum ranges in this scenario 
serves two purposes. First, currently fielded AHDs are mainly intended for land use. 
Varying the range of the AHD in this scenario explores the requirements needed over 
water. Second, this factor captures the performance of the AHD in varying conditions. 
For example, if the weather is very windy, thereby reducing the maximum effective range 
of the AHD, the data are still relevant for analysis. 

b. OD Maximum Range 

This is the maximum range of the OD. Like the AHD, this variable 
explores requirements and effectiveness given varying conditions. In this case, different 


states of visibility affect the maximum range of the OD. 
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c. WM Maximum Range 

This is the maximum range of the WM. As before, this variable explores 
requirements and effectiveness given varying conditions. 

d. AHD Firing Rate 

This defines the suppression duration of the AHD. Since its potential use 
ranges from immediate audio warnings to deliverable voice messages, this variable 
represents the time needed to employ the AHD against a target. For example, a longer 
firing rate represents the total time needed to broadcast a verbal message. 

e. Color Vulnerability 

The blue ship discriminates against inbound targets based on target 
kinematics, as discussed in Chapter 2 (Section E.3). This factor defines the suspicion 
level assigned to each category in the hierarchy of threats. Category I threats (target 
groups with three or more vessels traveling faster than 18 knots) create the highest level 
of suspicion, and in PYTHAGORAS terms, are assigned the highest color vulnerability. 

f OD Red Radius Ratio 

This defines the suspicion level needed against a target before it is eligible 
to be engaged by the OD. This level will always be a proportion of the WM red radius. 

g. WM Red Radius Ratio 

This defines the suspicion level needed against a target before it is eligible 
to be engaged by the WM. This level will always be a proportion of the Lethal red 
radius. 

h. Lethal Red Radius 

This represents the suspicion level threshold. Once this level is reached, a 
target is identified as hostile and is immediately engaged by lethal force. 

2. Target Factors 

a. Target Instance 

This is the number of targets grouped together for a particular threat type 
and location. This factor applies to the two hostile threat types and both neutral and 
suspect loitering types. For the neutral targets which represent an outbound vessel in the 
channel and a ferry crossing the channel, these threat categories are fixed at only one 


agent through all iterations. 
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b. Target Speed 

This is the average speed of the targets within a threat class. This factor 
applies to all non-loitering threats. 

c. Target Attribute Vulnerability 

This factor affects the deterrence level of a target. Since neutral targets 
are more likely to be influenced by non-lethal capabilities, they assume a higher 
vulnerability than suspect and hostile targets. The deterrence level becomes a function of 
the target’s attribute vulnerability and how often non-lethal capabilities are employed 
against them. 

d. Target Alpha Less Than Value 

This factor captures the deterrence level threshold within the 
PYTHAGORAS model. Once a target’s attribute level, i.e. deterrence level, depreciates 
below this setting, a target will deter. Within a single run, like threats will react the same. 
For example, in a single run, all neutral targets will have the same deterrence level 
threshold regardless of location or speed. 
D. DESIGN OF EXPERIMENT 

This thesis seeks to utilize a space filling design in order to capture all possible 
reactions of threat targets and effects of the non-lethal capabilities. A space filling design 
refers to thorough coverage of the many possible combinations of initial inputs. This 
fidelity allows the analyst to determine which factors have the greatest effect on the 
outcomes with a high level of confidence. 

To completely cover a design space, an analyst could use a gridded or full 
factorial design. These designs consider every possible combination of variable input 
parameters. For this research, there are a total of 33 input variables of interest. If the 
scenario explored only two levels (high and low) per variable, the simulation would 
require 2°> (# Levels” 8), or 8,589,934,592, design points. If each design point is 
iterated a minimum of 30 times for statistical validity, this design would take 4,903 
centuries to execute. Table 3 illustrates the number of design points required of different 


gridded designs. 
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Table 3. | Number of Simulation Runs Required in Various Gridded Designs (From 
OA4333, 2006) 


One alternative is to reduce the number of input variables to be explored. 
However, another alternative which allows for the same amount of input variables with 
even more levels per variable is to use a Nearly Orthogonal Latin Hypercube (NOLH) 
space filling design. Developed by LtCol Thomas Cioppa, U.S. Army, at the Naval 
Postgraduate School in 2002, the NOLH facilitates efficient exploration of a large design 
space by arranging factors in an uncorrelated manner (see also Cioppa and Lucas, 2006). 
Applied to this research, the NOLH allows for all 33 factors to be explored on various 
levels (up to 40) in only 257 design points. Keeping the factors as uncorrelated as 
possible simplifies the task of linear regression modeling because it makes it easy to 
separate the impacts of different input factors on the MOEs. The strongest correlation in 
the NOLH developed for this research is -0.08105, which is quite low. Figure 12 
depicts a scatterplot matrix of the correlations between the variable inputs used in this 
research. Each tiny sub-plot is a scatterplot of the levels of two different variables. The 
figure demonstrates both the space filling properties for quantitative factors of the NOLH 


and the lack of correlation between the factors. 
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Figure 12. Scatterplot Matrix of Factor Correlations 


E. DISTRIBUTED SIMULATION 

In addition to using efficient DOEs, another way to reduce required run-time of a 
simulation is to network computers together to distribute different parts of a computing 
task across individual processors (Law and Kelton, 2000). This thesis utilizes distributed 
simulation in applying the NOLH DOE to the baseline scenario. 

1. Maui High Power Computing Center Simulations 

Included in the Project Albert suit of tools is access to the Maui High Power 
Computing Center (MHPCC), which offers the ability to data farm over a large design 
space in a short amount of time. Using the Tiller, a program developed by Referentia 
Systems’ Steve Upton, the PYTHAGORAS scenario file and a study file are generated in 
eXtensible Markup Language (XML). The study file dictates what factors are to be 


varied and enfolds the design of experiment to be executed. MHPCC takes these files, 
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partitions the work over several computers, and runs the experimental design. Once the 
job is complete, MHPCC personnel zip the output and e-mail it back to the user in one 
consolidated file. This leads to significant time savings over other methods (Michel, 
2006). 

a. Debugging Experiment 

The initial debugging design included the factors from Table 2 with two 
exceptions. First, the six ‘attribute vulnerability’ factors were not included (at the time, 
an earlier version of PYTHAGORAS was being used that did not include these modeling 
capabilities). Second, the hierarchy of threats had not yet been built into the scenario. 
Therefore, instead of three color vulnerability factors (applied to threat categories I-III), 
there were four factors (applied to the two hostile and two neutral threat classes). With 
23 total factors, the NOLH spreadsheet tool, created by Dr. Susan Sanchez 
(NOLHDesigns.xls, 2006), generated a NOLH design yielding 257 design points or 
excursions. Using the MHPCC, each excursion was replicated 30 times, yielding 7710 
data points. This initial design provided insight into the inner workings of the model, the 
NOLH, and the interface with MHPCC. Several scenario enhancements resulted from 
the debugging design. 

b. Exploration Experiment 

Prior to the exploration experiment, Northrop Grumman released the 
current version of PYTHAGORAS (Version 1.10). Included in this update was the 
attribute modeling capability (see Pythagoras manual (Bitinas et al., 2006) for complete 
list of updates and new feature descriptions). Table 2 depicts all 29 factors applied to the 
NOLH spreadsheet tool for this design. 


Additionally, a technique called ‘lockstepping’ was used to alter several 
factors and create three new factors. Lockstepping involves creating input parameters for 
a variable from a derivative of another variable’s input parameters. For the final design, 
three lockstepping processes were applied: 


e The ‘OD red radius ratio’ and the ‘WM red radius ratio’ both represented 
proportions of other factors. For input into the PYTHAGORAS model, 
both were converted back to integer values. 
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e The two hostile and two neutral target color vulnerability inputs were 
derived from their respective instance, speed, and category-based color 
vulnerability input values. 


e The second hostile target group’s ‘alpha less than value’ was lockstepped 
to equal the first hostile target’s input value. Similarly, the second neutral 
target and the neutral fishing target’s ‘alpha less than value’ were 
lockstepped to equal the first neutral target’s value. 


The iteration level for each excursion was increased to 100, yielding 
25,700 data points. Although this design produced plenty of data and some interesting 
results, not all of the MOEs could be extracted with enough fidelity to answer the 
research questions. The PYTHAGORAS model allows for two main types of output from 
each simulation run: 1) end-of-run MOEs and 2) time-series MOEs. Due to the high 
volume of data produced with the time-series MOEB’s, this time-series MOEs are not 
available when using MHPCC. The end-of-run MOEs produced with this design using 
MHPCC are suitable for initial analysis with all threats combined into a single group. 
However, this research also calls for further analysis of each individual threat class. 
Since time-series MOEs are available when conducting a batched run locally, this author 
decided to conduct a second set of experiments in order to capture the remaining MOEs 
of interest. 

Ze Locally Paralleled Simulations 

Within the PYTHAGORAS directory is a file called PythEng.bat. This file 
allows the user to setup and execute multiple simulation runs from the command line of a 
local computer (Bitinas, 2006). The PythEng.bat file allows the user to output the time- 
series MOE files in addition to the end-of-run MOE files. 

A scripting program written in Ruby, an object-oriented programming language, 
automated execution of the PythEng.bat across each design point (see Appendix B). The 
decision to use Ruby vs. other object-oriented languages was based on its availability and 
compatibility. Ruby is open source and freely available for both development and 
deployment. Ruby in not restricted to a single platform or vendor, but can be run under 
Unix or Linux, Microsoft Windows, or specialized systems such as BeOS (What is Ruby, 


2006). 
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a. Test Experiments 
The test experiment consisted of running the first three design points with 
two iterations. The purpose of this small experiment was to validate the sequence and 
execution of the Ruby script designed for this simulation. After debugging for errors, the 
experiment was run a second time using two computers, ten design points (or excursions) 
each computer, and 100 iterations each excursion. In addition to successfully executing 
the Ruby scripts, this experiment provided initial run-time performance for optimal 
allocation of work during larger runs. 
b. Final Experiment 
The full design of 33 factors, 257 design points, and 100 iterations per 
design point was run on the same two computers. One computer handled 100 design 
points while the second, newer, computer tackled 147. After approximately three days of 
continuous run-time, the complete output detailing the MOEs on each individual target 
vessel was available and ready for analysis. 
3. Data Extrapolation 
When using the time-series MOEB’s, a separate file is generated for every run. 
Using VBA code (see Appendix C) and Microsoft Excel, all 25,700 individual files were 
compressed and concatenated into one composite data set. 
F. SUMMARY 
The experimental designs described in this chapter, used in conjunction with the 
computing power of distributed simulations, yield a large amount of detailed information 
from the simulation model. By implementing this analysis methodology, the resultant 
output is directly applicable in answering the research questions. The next chapter 


explores the analysis of this output, and what insight it brings to the questions at hand. 
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IV. DATA ANALYSIS 


For in this modern world, the instruments of warfare are not solely for 
waging war. Far more importantly, they are the means for controlling 
peace. Naval officers must therefore understand not only how to fight a 
war, but how to use the tremendous power which they operate to sustain a 
world of liberty and justice.* 


Admiral Arleigh Burke 


A. OVERVIEW 
From scenario development to implementation of the simulation experimental 
design, the previous chapters have focused on the generation of data for useful analysis. 
With a huge volume of data now available, it is time to see what insight this methodology 
produces. This chapter briefly describes the tools used for analysis, followed by a 
detailed analysis of each MOE and threat class. 
B. DATA ANALYSIS TOOLS 
The primary software used for data analysis is JMP Statistical Discovery 
Software™. In addition to its robust statistical applications and user friendliness, JMP 
dynamically links statistics with powerful graphic visualizations (JMP, 2006). 
1. Regression Models 
a. Mixed Stepwise Regression 
With 33 input variables, generating a model including interactions for a 
given MOE can be tedious. Therefore, stepwise regression is used as an approach to 
selecting a subset of the input variables when there is little theory to guide in the selection 
of terms (Stepwise Regression, 2006). The type of stepwise regression selected for this 
research is mixed, where the statistical software package alternates forward and backward 


stepwise regression until the remaining terms are significant. 


SArleigh Burke, (1961). Famous Navy Quotes. Retrieved September 4, 2006 from 
http://www.history.navy.mil/trivia/trivia02.htm. 
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b. Standard Least Squares Linear Regression 

After stepwise regression is used to select a model of interest, the model is 
fit to a linear regression using standard least squares. Once fit, three statistics are 
examined to determine the goodness and applicability of the model. These are the 


adjusted R’, the F-test statistic, and Student’s t-test statistic. 


The adjusted R* estimates the proportion of variation in the response 
variable that is explained by the predictor variables and not random ‘noise’ or error. The 
purpose in using the adjusted R’ instead of the regular R* value is to make the model 
more comparable against models with a different number of predictor variables, and 


therefore different degrees of freedom (A Multiple Regression, 2006). 


An Analysis of Variance (ANOVA) of the data is performed to determine 
if the relationship between the predictors and the response variable is statistically 
significant. The resultant F-test statistic and its associated p-value reveal, for a defined 
confidence level, that at least one of the predictors explains the response variable better 
than the null model. For this and all follow-on analysis, the confidence level selected for 
ANOVA analysis is 95%, meaning that when the F-test p-value is less than 0.05, the 


model is statistically significant. 


Since the F-test does not specify which of the predictor variables is 
significant, a Student’s t-test is performed on each of the variables. This determines if 
the variable is statistically significant, within a confidence level, in the presence of all 
other predictor variables. The confidence level selected for t-tests conducted in this 
research is 90%. Therefore, when the t-test p-value is less than 0.10, the associated 
predictor variable is determined to be statistically significant. 

2. Recursive Partitioning 

Recursive partitioning refers to the partitioning of data according to a relationship 
between the input variables and the output variable, creating a ‘tree’ of partitions. 
Recursive partitioning is not necessarily predictive like linear regression, but is an 


effective data mining technique because (Recursive Partitioning, 2006): 
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e It is good in exploring relationships without having a good prior model; 

e It handles large, complex problems easily; and 

e The results are very interpretable. 

The simulation model for this research produces complex and highly variable 
data. Therefore, the recursive partitioning method effectively extracts critical knowledge, 
insight and trend analysis in answering the research questions where traditional 
regression falls short. 

3. Data Visualization 

Several graphical visualizations are used to compliment and highlight the 


statistical results. The following describes the primary visualization tools used in this 


research: 

° Interaction Profiles — In an interaction profile, non-parallel lines 
demonstrate interaction between two variables. This provides a clear 
visualization of both the intensity and relationship of the interactions; 

e Pairwise Plots — Pairwise plots are used frequently to display various 
relationships between two variables; and 

e Partition Trees — Partition trees display the recursive partitioning as a 


decision tree with associated statistics. 

c. OVERALL ANALYSIS 

The following analyses explore the three main MOEs for all threat types 
combined. 

1. MOE I - Deterrence Ratio 

When calculating this MOE from the generated data, the hostile threat classes are 
not included. Since the hostile targets are modeled as suicide threats, ‘deterrence’ of a 
hostile threat is only a temporary state before its attack is resumed. Therefore, no hostile 


threats will have an ending status of deterred. 


With the deterrence ratio defined as the response variable, stepwise regression is 
used to identify the predictor variables of interest. The factors eligible for the stepwise 
process are those that represent controllable factors. These included ranges and firing 


rates of the non-lethal capabilities and the ROE postures. Details about the resultant 
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model from the stepwise regression, performed on these seven factors, all two-way 
interactions between them, and all quadratic terms of the main effects, are obtained via 


the least-square linear regression module. 


Summary of Fit Parameter Estimates 

RSquare I a Term Estimate StdError tRatio Prob=tt 
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Figure 13. Final Linear Regression Model on Overall Deterrence Ratio 


Figure 13 displays the statistics and model performance values obtained from the 
linear final regression model. Number | in Figure 13 reports an overall adjusted R’ of 
26.3%. This means that only one quarter of the variability observed in the deterrence 
ratio is explained by the predictors selected for this model. A relatively low R’ in a 
model that represents a complex and highly variable real word scenario is not surprising. 
Number 2 in Figure 13 shows, despite the low R’, the regression model is statistically 
significant given the F-test p-value. The section labeled ‘Parameter Estimates’ in Figure 
13 lists the final factors selected as predictor variables for the model and the associated t- 
test statistic and p-value for each predictor. Of particular interest is the t-test conducted 
on OD range. Number 3 in Figure 13 shows OD range alone is statistically insignificant 
in affecting the deterrence ratio. However, a critical element of this research is to explore 
the effectiveness of the non-lethal capabilities when used as an element in a system of 
capabilities. Number 4 in Figure 13 exemplifies that although OD range alone is 


statistically insignificant, the interaction between OD range and the ROE level for 
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employing WMs is critical. Number 5 in Figure 13 shows that the quadratic terms for six 
of the seven main effects are statistically significant. Despite the statistical significance 
of the quadratic terms, including quadratic effects as potential explanatory terms had little 


impact on the model’s explanatory power. 


Of the 26 predictor terms selected for the model, 19 are statistically significant 
interaction terms. These relationships are more clearly visualized using JMP’s 
interaction profiler, as seen in Figure 14. The former example of interaction between OD 
range and the ROE level for WM employment is highlighted by Number | in Figure 14. 
This shows that the effect of WM ROE is smaller when the value for OD max range is 
low, but has more effect on the overall deterrence ratio for higher values of OD max 
range. Figure 2 displays an opposite relationship between WMRedRad and 
LethalRedRad. Here, the restrictiveness of WM ROE has a greater impact on the overall 
deterrence ratio when there is a freer ROE for lethal force. When ROE for lethal force is 
highly restrictive, the ROE level for WM employment has little effect on the overall 


deterrence ratio. 


Interaction Profiles 
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Figure 14. Interaction Profiles of Predictor Variables 
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In addition to the information provided in Figure 13, other pairwise plots are 
available to assess the performance of the selected regression model. Figure 15 plots the 
fitted deterrence ratio against the model residuals. A primary assumption underlying 
linear regression is that the residuals are homoscedastic, meaning that the residuals are 
equally varied across the observed values of the response variable. If strong ‘fanning’ 
trends are detected in this plot, then there is evidence of heteroscedasticity, and the 
assumption of equal variance in the residuals is not met. Figure 15 shows that there is no 
strong evidence of heteroscedasticity (the vertical spread between residuals is consistent 
across the range of predicted deterrence values) and the modeling assumption is not 


violated. 


Residual by Predicted Plot 
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Figure 15. Plot of Residuals vs. Fitted Variables 


Another assumption of linear regression modeling is normality in the distribution 
of errors. Figure 16 portrays the use of visual plots to determine the normality of the 
residuals. The Normal Quantile plot is used to investigate whether the data exhibits the 
standard bell curve of a normal distribution. Since the plotted deterrence rate points 
closely follow a linear pattern in the Normal Quantile plot, the assumption of normality is 


not violated. 


36 


Residual Overall Det 


001.01 .080.25 50 75.905 .99,858 
' , aa 





ee ees ee 


Normal Quantile Plot 





Figure 16. Plot of Residuals of Overall Deterrence Data 


Although the linear regression provides clear insight on the importance of the 
interaction between input variables, the large, complex scenario is not otherwise modeled 
well by linear regression. Therefore, we use recursive partitioning on the data. Unlike 
the linear regression model, the partition tree for this MOE incorporated all 33 input 
factors—both the controllable factors (non-lethal capabilities ranges and firing rates) and 
the uncontrollable factors (hostile and neutral behaviors). The first step is to recursively 
‘split? the data to create a partition tree. For each split, the overall value of R? is 
examined. From this information, the point of diminishing returns was used to define the 
appropriate number of splits. The same method is used to determine splits for all 


subsequent recursive partitioning. 


Using four splits, the partition tree for overall deterrence rate yields an R° of 
31.4%. Although this model captures more variability than the linear regression, the 
nature of the scenario continues to produce relatively low R’ values. Figure 17 shows the 
partition tree representing the recursive partitioning for this MOE. Number | in Figure 
17 shows that the most dominate factor in affecting the deterrence rate is the range of the 
AHD. When AHD range is greater than 975 yards (51 pixels), the overall deterrence 
ratio is 66%, a 15% increase over when the AHD range is less than 975 yards. Figure 2 
shows that if 1) the AHD Range is less than 975 yards and 2) the channel transiting 


neutral target has a stubborn deterrence level threshold, then the firing rate of the AHD is 
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the next factor to affect the deterrence ratio. When each AHD broadcast takes less than 
54 seconds (firing rate is greater than .28 shots/time step), the mean deterrence rate is 


47%, vs. only 15% when the broadcast takes longer. 























































































































Ss 
All Rows 
Count 25500 
Mean 0.5901623 
Std Dev 0.2274842 
[ ] 
AHDRange<51 AHDRange>=51 
Count 11800 Count 13700 
Mean 0.5071342 Mean 0.6616755 
Std Dev 0.2526044 Std Dev 0.1741063 
TNAIpha<235 TNAIpha>=235 TSFinst>=2 TSFinst<2 
Count 3300 Count 8500 Count 11300 |] Count 2400 
Mean 0.3438094 Mean 0.5705427 Mean 0.6306014 |} Mean 0.8079828 
Std Dev 0.2657229 Std Dev 0.2163299 Std Dev 0.1634146 Std Dev 0.1461473 
I 
AHDFirRate<0.28 AHDFirRate>=0.28 
Count 1300 Count 2000 
Mean 0.1528298 Mean 0.4679462 
StdDev 0.197754 Std Dev 0.2279862 
Figure 17. Partition Tree on Overall Deterrence Ratio 


Given these findings, the primary insight gleaned from this MOE is that the AHD 
has the most impact in deterring targets. As this is the initial non-lethal capability used in 
the tactic employed in this scenario, this finding is generalized to say that the first 
response capability has the most affect in deterring targets. 

2. MOE II - Hostile Identification Ratio 

As before, stepwise regression is used to select the predictor variables of interest 
given hostile identification ratio as the response variable. Number | in Figure 18 shows 
an adjusted R* of only 6.7%. Again, although the R’ is small, Number 2 shows that the 
linear regression model is statistically significant. Number 3 demonstrates that 
interaction and quadratic terms are equally as important in this MOE. As before, the 


assumption of homoscedasticity and normality appear reasonable. 
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Summary of Fit 




















Parameter Estimates 


RSquare 0 O65. Term Estimate StdError tRatio Probett| 
RSquare Adj 1 Intercept 0.5709612 0.00552 10343 0.0000 
Root Mean Square Error 0 a AHDRange -0.000592 0.000047 -1260 <.0001 
Mean of Response 0.5111 ODRange 0.0001303 0.000064 2.03 0.0421 
Observations (or Sum VWgts) 25700 VWiMRange 0.0005861 0.000128 457 =.0001 
AHDFirRate -0.123051 0.007828 -1572 <.0001 
Analysis of Variance LDRedRad 0.002061 0.000246 O84 0.4028 
Source DF Sumof Squares MeanSquare  F Ratio WMRedRad 0.0007038 0.000178 396 <.0001 
Model 30 26.80624 0.893541 63.0918 LethalRedRad -0.000078 0.000102 -0.77 0.4433 
Error 25669 363.53894 0.014163 Prog (AHDRange-52.5019)*(WiRange-15.0078) -0.000017 o.000009 1.98 0.0479 
C. Total 25699 390.34518 (AHDRange-52.5019)*(AHDFirRate-0.33502) -0.004423 0.000433 10.23 <.0001 
(SHDRange-52.501 9)*(LDRedRad-8.75097) 0.0000942 0.000012 8411 <.0001 

2 (AHDRange-52.5019)*(WMRedRad-16.6654) -0.000084 0.000009 -9.65 <.0001 

(AHDRange-52.501 9)*(LethalRedRad-32.5019) 0.0000601 0.000007 9.23 <,0001 

(ODRange-35.0156)*(/VMRange-15.0078) 0.0000662 0.000011 5.80 <.0001 

(ODRange-35.0156)*(AHDFirRate-0.33502) -0.005726 0.000711 6.06 <.0001 

(ODRange-35.0156)*(LDRedRad-8.75097) 0.0000348 0.000015 231 0.0209 

(ODRange-35.0156)*(VMRedRad-16.6654) -0.000042 0.00001 -4.20 «0001 

(VMRange-15.0078)*(AHDFirRate-0.33502) 0.0226291 0.001514 1495 <0001 

(VMRange-15.0078)*(LDRedRad-8.75097) 0.000124 0.000027 454 «0001 

(VVMRange-15.0078)*(VV¥MRedRad-16.6654) -0.000081 0.00002 -3.99 <.0001 

(WMRange-15.0078)*(LethalRedRad-32.5019) -0.000118 0.000016 -752 <.0001 

2 (AHDFirRate-0.33502)*(LDRedRad-8.75097) -0.015462 0.00193 -8.01 <.0001 

10 20 30 40 50 60 70 80 90 (AHDFitRate-0.33502)"(VWMRedRad-1 6.6654) 0.0117848 0.001291 9413 <.0001 

Overall Kill Predicted PO.0000 (LDRedRad-8.75097)"(WMRedRad-1 6.6654) 0.004643 0.000039 11.84 <.0001 
RSq=0.07 RMSE=0.119 (LDRedRad-8.75097)"(LethalRedRad-32.5019) -0.000242 0.000027 -893 <.0001 
(WMiRedRad-16.6654)*(LethalRedRad-32.5019) 0.0001865 0.000021 883 <.0001 

(AHDRange-52.5019)*(AHDRange-52.5019) -0.000016 0.000004 -4.61  <.0001 

(ODRange-35.0156)*(ODRange-35.0156) -0.000042 0.000007 6.05 <.0001 

(LDRedRad-8.75097)*(LDRedRad-8.75097) -0.000346 0.000028 -12.57 <.0001 

(WMRedRad-16.6654)*(VWMRedRad-16.6654) -0.00023 0.000019 -12.27 <.0001 

(LethalRedRad-32.501 9)*(LethalRedRad-32.5019) 0.0000918 0.000011 8.28 <.0001 


Figure 18. Final Linear Regression Model on Overall Hostile Identification Ratio 


The partition tree for this MOE sheds light on the extremely poor R’ in the linear 
regression model. Figure 19 illustrates the first three splits are all factors that the blue 
ship has no control over, to include hostile target speed and number of suspicious fishers. 


The predictor variables modeled in the linear regression model do not even appear in the 


partition tree until many successive splits later. 


TH2Spd<6 
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Figure 19. 
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Partition Tree on Overall Hostile Identification Ratio 


Number | in Figure 19 shows the best case scenario, where if: Hostile2 target 
speed is greater than 14 knots (6 pixels/time step), hostile target speed is greater than 12 
knots (5 pixels/time step), and the number of Suspect Fishers is greater than 2, then 


targets are identified as hostile 55% of the time. 


Number 2 in Figure 19 represents the worst case scenario. If Hostile2 target’s 
speed is less than 14 knots (6 pixels/time step), targets are identified as hostile 44% of the 


time. 


The difference between the best case and worst case scenario is only 11%. 
Combined with a lower R* of only 17%, the MOE selected does not appear to give much 
insight towards the research questions other than the mean hostile identification ratio. 
Underlying statistics such as misidentification rates are explored in subsequent analyses. 

3. MOE III — Warning Zone Identification Ratio 

The previous MOE only corresponds to a target identified as hostile, but not how 
it was identified. The scenario models two very different ways by which a target can be 
identified as hostile. This MOE captures those differences by recording the proportion of 
targets identified outside of the exclusion zone (given that a target is identified as 
hostile). The linear regression shows an R’ of 26.6% and a Statistically significant F-test 
p-value. As with the previous two MOE regressions, there is clear evidence that 
interactions and quadratic terms are critical. With respect to the modeling assumptions, 
the model is homoscedastic, but the residuals are not normally distributed. Instead, the 
residuals display a strong right skew. Although this assumption is violated, it does not 
affect the regression because the MOE selected is a mean; and therefore is not heavily 
dependent upon the distribution around the statistic. Figure 20 displays the model results 


in full detail. 
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Summary of Fit 








RSquare 
RSquare Adj 
Root Mean Square Error g. 
Mean of Response 0.082765 
Observations (or Sum Vgts) 25700 
Analysis of Variance 
Source DF Sumof Squares Mean Square F Ratio 
Model 29 15640682 5.39334 321.8836 
Error 25670 43011506 
C.Total 25699 586.52188 
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Parameter Estimates 


Term 
Intercept 
AHDRange 
oDRange 
WMRange 
AHDFirRate 
LDRedRad 
WMRedRad 
LethalRedRad 


0.01676 (SHDRange-52.5019)*(ODRange-35.0156) 
(AHDRange-52.501 9)*(WMRange-15.0078) 
(AHDRange-52.5019)*(AHDFirRate-0.33502) 


(AHDRange-52.5019)*(LDRedRad-8.75097) 
(AHDRange-52.5019)*(WMRedRad-16.6654) 
(AHDRange-52.5019)*(LethalRedRad-32.5019) 
(ODRange-35.0156)*(VWMRange-15.0078) 
(ODRange-35.0156)*(VwMRedRad-16.6654) 
(ODRange-35.0156)*(LethalRedRad-32.5019) 
(WMRange-15.0078)*(4HDFirRate-0.33502) 
(WMRange-15.0078)*(LDRedRad-8.75097) 
(VMRange-15.0078)*(VWMRedRad-16.6654) 
(WMRange-15.0078)*(LethalRedRad-32.5019) 
(SHDFirRate-0.33502)*(LDRedRad-8.75097) 
(SHDFirRate-0.33502)*(vMRedRad-1 6.6654) 
(LDRedRad-8.75097)*(LethalRedRad-32.5019) 
(AHDRange-52.5019)*(AHDRange-52.5019) 
(ODRange-35.0156)*(ODRange-35.0156) 
CAMRange-15.0078)*C/vMRange-15.0078) 
CSHDFirRate-0.33502)*(AHDFirRate-0.33502) 
(LDRedRad-8.75097)*(LDRedRad-3.75097) 
(VMRedRad-16.6654)*(VVWMRedRad-16.6654) 


Estimate 
-0.002473 
0.0014191 
0.0011119 
0.0002643 
0.3176941 
-0.006586 
-0.00143 
-0.00231 
0.0000306 
0.0000366 
0.0069645 
-0.000016 
0.0000628 
-0.000145 
0.0000382 
-0.000044 
-0.000074 
-0.023452 
-0.000135 
0.0001179 
-0.00007 
-0.005242 
-0.014568 
0,0002558 
-0.000046 
0.0000106 
-0.000185 
0.9519154 
0.0001919 
0.0001239 


Std Error 
0.006049 
0.000051 
o.00007 
0.000139 
0.008485 
0.000246 
0.000153 
0.000102 
0.000004 
0.000009 
0.000471 
0.000012 
0.000009 
0.000007 
0.000012 
0.00001 
0.000009 
0.001643 
0.00003 
0.000022 
0.000017 
0.002062 
0.001342 
0.00002 
0.000004 
0.000007 
0.000029 
0.115891 
0.000018 
0.000011 


t Ratio 
-0.41 
27.91 
15.95 
1.90 
37.44 
-26.80 
-9.35 
22.75 
6.98 
3.95 
14.78 
1.26 
6.82 
-20.42 
3.09 
-4.50 
-8.18 
14.27 
-4.55 
5.32 
“4.11 
-2.54 
10.85 
13.06 
-11.12 
1.42 
-6.49 
8.21 
10.40 
11.59 


Probett] 
0.6827 
<.0001 
<,0001 
0.0580 
<,0001 
<,0001 
<.0001 
<,0001 
<,0001 
<,0001 
<,0001 
0.2064 
<,0001 
<,0001 
0.0020 
<,0001 
<.0001 
<,0001 
<,0001 
<.0001 
<,0001 
0.0110 
<,0001 
<.0001 
<.0001 
0.1549 
<,0001 
<,0001 
<,0001 
<.0001 


Figure 20. Final Linear Regression Model on Warning Zone Identification Ratio 


Figure 21 shows the corresponding partition tree for this MOE. The dominant 
input factor is the speed of the hostile target. If hostile speed is less than 14 knots (6 
pixels/time step), targets are identified as hostile outside of the exclusion zone, using the 
non-lethal capabilities, 24.6% of the time. Otherwise, the mean identification rate outside 


the exclusion zone is only 3.8%—a substantial difference. 
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Figure 21. Partition Tree on Overall Warning Zone Identification Ratio 
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The linear regression models of all three MOEs consistently establish the 
significance of interactions between input factors. This illustrates the importance of 
modeling the non-lethal capabilities as a system and not performing separate analyses for 
each individual component. Otherwise, the linear regression models produce little else of 
importance. Therefore, the remaining analyses will focus on recursive partitioning and 
ordinary statistics. The partition trees provided initial insight as to what factors have the 
most impact given any threat faced by the blue ship. 

D. ANALYSIS BY THREAT CLASS 

The following analyses explore the MOEs applied to each threat class. This 
enables more fidelity in the analysis, resulting in more detailed findings in regards to 
non-lethal effectiveness. 

1. Hostile Targets 

The MOE of interest for specific analysis of the hostile threat class is the MOE 
IH, the warning zone identification ratio. Figure 22 displays the partition tree for the first 
hostile target class, which is the group that conducts the overt attack at scenario start. 
The R’ for this model is 81.6%, showing that most of the variability in the MOE is 
explained with only 4 input factors. The first split occurs when the target speed is 14 
knots (6 pixels/time step). This is consistent with the initial findings of MOE 3 using all 
threat classes. When looking specifically at this threat class, the impact of the inbound 
hostile speed is dramatic. Number | in Figure 22 shows the blue ship uses the non-lethal 
capabilities to identify the targets as hostile outside of the exclusion zone 73% of the 
time. However, if the inbound targets are moving faster than 14 knots, identification in 
the warning zone occurs only 6.5% of the time. This finding supports the research 


question of when non-lethal capabilities are tactically appropriate. 


Number 2 in Figure 22 shows the impact of the AHD firing rate when the target 
speed is less than 14 knots, but ROE employment of the OD is more restrictive. In this 
case, hostile identification outside the exclusion zone is improved by 52% (from 37% to 
89%) when the AHD broadcast takes less than 38 seconds (firing rate is greater than .39 
shots/time step). This provides valuable insight into the tactical implications when using 
the AHD as an immediate audio warning device versus as a verbal message delivery 
system. 
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Number 3 in Figure 22 exemplifies the dominance of ROE-type factors of the 
non-lethal capabilities factors. For both branches of the partition tree, the next significant 


factor after hostile target speed is ROE restrictiveness of the non-lethal employment. 












































































































































All Rows 
Count 
Mean 0.2078909 
Std Dev 0.3407328 
| 
THSpd>=6 THSpd<6 
Count 202 Count 55 
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Figure 22. Partition Tree on Hostile Identification Ratio — Hostile Target Class 


The second hostile target group represents the group conducting a more covert 
attack from a closer range on the blue ship. Like the first target group, the model has a 
reasonably high R° of 64.8%. Number 1 in Figure 23 tells of comparative differences 
between the two hostile threat types where the only difference between the two is 
geography. The first difference noted is the count of observations observed. Because the 
second hostile target group attacks the blue ship towards the end of the scenario, there are 
iterations when the blue ship reaches its objective point before the hostile targets 
intercept. This is primarily a function of when the hostile targets detect the blue ship and 
what the hostile speed setting is for that run. Therefore, the data for the partition tree 
have been pre-screened to include only those data points where an interaction between 


the hostile group and the blue ship occur. 


The second difference noted is that the mean identification ratio is 13.8%, which 
is 7% lower than that of the first hostile target group. As expected, this demonstrates that 
geography impacts the effectiveness of the non-lethal capabilities. By conducting a 
covert attack at a closer range, the blue ship has less reaction time. Number 2 in Figure 


23 shows that unlike facing hostile targets in a more open-water environment, the 
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dominating factor in constrained geography is the restrictiveness of the ROE. For the 


tactic modeled, a quicker escalation to the use of the OD yields a 25% difference in 


hostile identification outside of the exclusion zone. Number 3 in Figure 23 shows that 


the employment time of the AHD is critical, just as it was for successfully identifying the 
first hostile target group. 
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Figure 23. 


Partition Tree on Hostile Identification Ratio — Hostile2 Target Class 
2. Neutral Targets 


The MOE of interest for analysis of the neutral threat class is the MOE I, the 
deterrence ratio. Like the second hostile threat group, the data were pre-screened to 
analyze only data points where an interaction between the neutral target and the blue ship 
occur. Figure 24 displays the partition tree for the channel transiting neutral target. The 
probability of deterrence for the neutral target is 92.3%. When the neutral target’s speed 
is less than 16 knots (7 pixels/time step), the neutral target is deterred 100% of the time. 


Although the regression tree does not show dominant effects of the non-lethal 


capabilities, this model demonstrates that positive action, through non-lethal 


employment, produces very high deterrence rates in neutral targets (regardless of the 
specific requirements). 
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Figure 24. Partition Tree on Deterrence Ratio — Neutral Target Class 


In further examination of the data, there are 27 instances of 20,037 runs where the 
neutral target crosses within the blue ship’s exclusion zone despite being deterred. All 27 
points originate from the same design point, meaning that given 100 iterations of the 
same inputs, 27 runs resulted in this outcome. Reasons for this include maneuvering 
constraints of the neutral target and the timeliness of the target’s deterrence. There are 
also four instances of 20,037 runs where the neutral target is identified as hostile outside 
of the exclusion zone. All four points originate from the same design point of input 
variables. A replay of these four runs using PYTHAGORAS’ playback feature showed 
that the hostile identification event and the deterrence event happened at virtually the 
same time. Of the 21,569 data points, the blue ship identified the neutral target as hostile 
only 0.5% of the time (118 times). These cases were generally a result of the neutral 
target’s continued negative response to non-lethal employment and an unwillingness to 


deter. 


The second neutral target, representing a crossing ferry in a geographically 
constrained area, also exemplifies the effects of geography on the effectiveness of the 
non-lethal capabilities. Figure 25 shows a probability of deterrence of 82.7%, almost 
10% less than that of the first neutral target. Number 1 in Figure 25 shows that the factor 
with the most effect on the deterrence rate is the range of the AHD. When the AHD 
range is greater than 973 yards (55 pixels), deterrence probability is 94%. However, for 
lesser ranges of the AHD, the probability is reduced to 72.7%. This result is consistent 
with the overall analysis of MOE 1, showing that the requirements of the first response 
capabilities are critical. This also brings specific tactical insight into the use of the AHD. 


If the weather supports AHD max effective range past 1,000 yards, the AHD is a very 
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effective first response non-lethal capability. However, if performance range of the AHD 
is degraded due to inclement weather conditions, such as high winds, the AHD is not as 


effective as a first response capability. 
| 
























































All Rows 
Count G"2 Level Prob 
21428 19702.227 0 0.1724 
1 0.8276 
AHDRange>=55 AHDRange<55 
Count G"2 Level Prob Count G'2 Level Prob 
10169 4675.6243 0 0.0611 11259 13201.074 0 0.2730 
1 0.9389 1 0.7270 
TN2ColVul>=0.7 TN2ColVul<0.7 
Count G2 Level Prob Count G2 Level Prob 
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TN2Spd<7 TN2Spd>=7 
Count G"2 Level Prob Count G*2 Level Prob 
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1 0.7642 1 0.0881 

















Figure 25. Partition Tree on Deterrence Ratio — Neutral2 Target Class 


Like the first neutral target, further examination of the data is conducted. There 
are 460 instances where the neutral target crosses within the blue ship’s exclusion zone 
despite being deterred. This is respectably more than the 27 observed with the first 
neutral target and is attributed to the maneuvering restrictions of the second neutral 
target. Despite being deterred, the ferry neutral often has no where to run to if caught 
between the blue ship and land. There are 14 instances where the neutral target is 
identified as hostile outside of the exclusion zone, also all from the same design point. 
As before, a random sampling of these 14 instances showed that the hostile identification 
event and the deterrence event happened at virtually the same time. Of the 25,123 data 
points, the blue ship identified the neutral target as hostile only 0.7% of the time. 

3. Loitering Targets 

The final analyses explore the mix of two threat groups loitering in the same 
geographic area, specifically a choke point. Figure 26 shows the partition tree of the 
deterrence ratio of the neutral fishing targets. Number | in Figure 26 displays the first 
split in this partition. Unlike previous analyses, the factor with the most effect is the 
deterrence threshold of the neutral fishers. Number 2 in Figure 26 shows when neutral 


fishers have high deterrence level thresholds, the factor of most significant impact is 
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again AHD range. With these slower moving targets, the performance threshold of the 
AHD is 750 yards (43 pixels). At ranges greater than 750 yards, the deterrence rate is 
77%, compared to only 37.5% for lesser ranges. Number 3 in Figure 26 shows when 
AHD range is less than 750 yards, the next critical factor is the employment time of the 
AHD. In this case, deterrence rate is improved 49% (from 11.8% to 61%) when the 
AHD broadcast takes less than 54 seconds (firing rate is greater than .28 shots/time step). 
The last two findings continue to emphasize the importance of the first response 


capability against neutral targets. 
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Figure 26. Partition Tree on Deterrence Ratio — Neutral Fisher Target Class 


14.4% of the neutral fishers that are deterred cross within the blue ship’s 
exclusion zone despite being deterred. This is largely due to the even more restrictive 
geography combined with slower moving targets. There are no instances where the 
neutral fisher is both deterred and identified as hostile outside of the exclusion zone. Of 
the 25,700 data points, the blue ship identifies the neutral fisher as hostile outside the 


exclusion zone only 0.1% of the time. 


Figure 27 shows the partition tree of the deterrence ratio of the suspect fishing 
targets. Like the neutral fishing targets, Number 1 in Figure 26 shows the factor with the 
most effect is the deterrence threshold. Number 2 depicts that when the suspect fishers 
are easily deterred, the speed of the first response capability is more important than the 


range. This is expected since the suspect fishers do not attack the blue ship until well 
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within the max range of the AHD. For this situation, deterrence rate is improved 40% 


(from 10% to 50%) when the AHD broadcast takes less than 42 seconds (firing rate is 
greater than .36 shots/time step). 





















































































































































All Rows 
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Mean (0.020322 Mean —0.2435198 
StdDev 0.079467 Std Dev 0.3046688 
AHDRange<74 AHDRange>=74 AHDFirRate<0.36 AHDFirRate>=0.36 
Count 191 Count Count Count 
Mean 0.0105536 Mean 0.0894245 Mean 0.0985025 Mean 0.5024793 
StdDev 0.0460924 StdDev 0.1775414 StdDev 01251468 StdDev 0.3602141 
THColVul<0.82 THColVul>=0.82 TNColVul>=0.64 TNColVul<0.64 TSFinst>=2 TSFinst<2 
Count 18 Count 9 Count 19 Count 6 Count 9 Count 5 
Mean —0.0234722 Mean —0.2213292 Mean —0.0539274 Mean —0.2396569 Mean —0.2965395|| Mean —0.8731709 
Std Dev _0.0861923 StdDev 0.2393651 StdDev 0.073804 StdDev 0.1552647 Std Dev 0.262542|| StdDev _0.1290504 


























Figure 27. Partition Tree on Deterrence Ratio — Suspect Fisher Target Class 


Overall, the suspect fishers penetrate the blue ship’s exclusion zone 95.4% of the 
time. When looking at MOE 3, the blue ship identifies the suspect fishers as hostile 
outside of the exclusion zone only 4 times of 25,065 observations. Combined with the 
results of the analyses of the neutral fishers, two very important conclusions are drawn. 
First, based on the deterrence rate of the neutral targets and the hostile identification rate 
of the suspect targets, the application of non-lethal capabilities is extremely effective in 
separating threats from non-threats in a homogenous group of threats. Second, based on 
the percentage of both neutral and suspect fishers that penetrate the exclusion zone, the 
counter-personnel non-lethal capabilities modeled in this research are not sufficient when 
used alone in order to deter loitering targets in a chokepoint. 

E. SUMMARY 
In this chapter, application of validated statistical tools has been demonstrated to 
successfully conduct data analysis on several MOE and threat class combinations. The 


primary findings discovered from the analysis are as follows: 


e The employment of non-lethal capabilities is extremely effective when 


used to identify threats from non-threats in an ambiguous situation. 


Inbound speed is the critical factor in identifying and engaging inbound 
hostile threats outside of the exclusion zone. 


The number of inbound targets has little to no impact on identification and 
engagement rates of hostile targets. 
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e The first response non-lethal capability is the most crucial in deterring 
non-suicidal targets. 


e The AHD is significantly more effective when employment time is less 
than 30 seconds against hostile targets and 1 minute for neutral or loitering 
targets. 

° When used alone, counter-personnel non-lethal capabilities fail to deter 


loitering targets who attack when within close proximity 
The following chapter links these findings back to the primary research questions 


and discusses their impacts. 
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V. CONCLUSIONS AND RECOMMENDATIONS 


Now this is not the end. It is not even the beginning of the end. But it is, 
perhaps, the end of the beginning.** 


Sir Winston Churchill 


A. RESEARCH SUMMARY 

This research began with questions. It is time to return to those questions, tie the 
corresponding analysis findings to them, and determine what insight can be gained. The 
research questions originally stated in Chapter 1 are reproduced below, but now followed 


by the applicable results obtained from this research. 


What non-lethal capabilities are required in a maritime force protection 
environment in order to effectively determine intent and/or deter suspicious small 
vessels? This research explored this question against a multitude of threat-types, 
different geographies, and varying conditions. As such, no singular answer is 
appropriate. Instead, multiple insights are gained for all possible environments. 
Therefore, this primary question is addressed in each of the follow-on discussions to the 


sub-questions below. 


In what region of conditions are non-lethal capabilities tactically appropriate ? 
The effectiveness rates of identification and deterrence are highly sensitive to the 
application of the first response capability and the timeliness in employment of 
subsequent capabilities. If the speeds of the incoming targets are less than 14-16 knots, 
the tactic modeled is effective. A 39% increase in effectiveness is observed in the worst 
case scenario (suicidal hostile target in a geographically constrained region) when the 
employment time of the AHD is kept below 30 seconds. When inbound speeds are 


higher, alternate tactics and capabilities must be explored. 


What, if any, are the major differences given varying geographic constraints? 
For both hostile and neutral targets, constrained geography resulted in lower 


identification and deterrence rates. Any warfighter knows that limited geography 


**Winston Churchill, (1942). Classic Quotes. Retrieved September 7, 2006 from 
http://www.quotationspage.com/quote/24921.html. 
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translates into reduced prosecution time. The specific insight gained through this 


research for a constrained geography environment is: 


e The effectiveness of the AHD must exceed 975 yards. 

e Effectiveness is extremely dependent on a relaxed ROE for the OD and 
WM. 

e If these two conditions cannot be met, a different tactic is required. 


What requirements are needed of non-lethal capabilities in order to effectively 
separate threats from non-threats in a homogeneous group? The first necessary task 
when facing a group of mixed targets is to separate the threats from the non-threats. 
Within the loitering targets, the neutral targets were identified as non-threats 74% of the 
time while the suspect fishers were recognized as threats 95.4% of the time. The most 
critical factor in achieving these levels of effectiveness was simply the use of the non- 
lethal capabilities in accordance with the modeled tactic. Meaning, the specific 
requirements of the non-lethal capabilities did not cause drastic variations in the results. 
The fact that proactive non-lethal force was applied, regardless of specific ranges and 


firing rates, proved extremely effective in separating threats from non-threats. 


Conversely, the sole use of counter-personnel non-lethal capabilities in a choke 
point filled with multiple threat types did not prove effective for deterrence. Neutral 
fishers were deterred 80.6% of the time, however this dropped significantly to 37.5% 
when the AHD range was less than 750 yards. A critical finding was that if the AHD 
range is less than 750 yards, the firing rate should be kept below one minute in order 
produce the highest possible deterrence rate. Deterrence of the suspect fishers, which are 
the vital vessels to discourage, occurred only 5.4% of the time. Not surprisingly, this 
result mandates the necessity of picket boats and counter-materiel non-lethal capabilities 


for this environment. 


Ts a multi-agent simulation (MAS) an appropriate means to represent and analyze 
a ship’s ability to determine intent and use continuum of force? Yes. The flexibility 


inherent in MAS, and particularly within the PYTHAGORAS model, has provided the 
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framework necessary to answer the previous questions. PYTHAGORAS’ capacity to 
quantify and model elements such as ROE, fear, aggressiveness and influence, has been 
crucial in producing useful and applicable insight for a complex problem. 
B. POTENTIAL FOLLOW-ON RESEARCH 

As shown in this research MAS, efficient experimental designs and high power 
computing are appropriate for a broad scope of analysis in the realm of non-lethal 
capabilities and maritime force protection. The following recommendations are provided 
for areas of research which could benefit from the data farming methodology applied to 
this thesis. 

1. Requirements 

This thesis identified situations when targets are not effectively deterred using 
only counter-personnel non-lethal capabilities. Current doctrine employs small boats to 
be used as pickets to physically deter inbound vessels. Additionally, counter-materiel 
non-lethal capabilities are also being developed in the maritime environment. This 
research successfully modeled non-lethal capabilities modeled as a_ system. 
Recommended future efforts would be the expansion of this system using the same 
scenario to include counter-materiel capabilities and picket boats. 

2. Tactics 

One specific tactic, adapted from current Navy tactics, was modeled in the 
scenario. This research identified performance conditions in which the modeled tactic 
was not effective. The methodology used in this research can easily be used to conduct a 
comparative analysis of multiple tactics. By creating a set of scenarios where each 
represents a unique tactic, the scenarios can be compared to determine the most effective 
tactic(s) for each situation. 

2 Vulnerability Assessments 

As demonstrated, geography is a critical factor in the force protection of the blue 
ship. This methodology allows a port or chokepoint to be exhaustively searched to 
determine the most sensitive geographical points of vulnerability. Since real world maps, 


charts or satellite photos can be overlaid into the PYTHAGORAS model, construction of 
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a vulnerability-based scenario for any place in the world can be accomplished quickly. 
Future work should include application of this methodology to assist planners in port 
integrated vulnerability assessments prior to a ship’s arrival. 

4. Alternate Mission Objectives 

The use of non-lethal capabilities is by no means limited to an entering port force 
protection scenario. The application of this methodology would be extremely relevant 
for analysis of alternate missions, such as straits transits and maritime interdiction 
missions. 

5. Refined Input Parameters 

As seen in the deterrence of the neutral fishers, the most significant factor was the 
deterrence level threshold, or willingness of the targets to deter. Since no data exists on 
these values, the experiment ranged across a wide variety of possibilities. As non-lethal 
capabilities become fielded, and real-world data is collected on these behavioral 
variables, future analyses can incorporate more refined behavior inputs or explore the 
reasonableness of these input parameter ranges. 
C. RESEARCH IMPACT 

Using the data farming methodology to answer the research questions outlined in 
this thesis is only the first step. The next level of insight is reached by understanding the 
applicability of this research. The following outlines the impact of this research and 


follow-on initiatives in the fielding of non-lethal capabilities for maritime missions. 


Focuses Capability Requirements - This research uses analytic rigor to 
compliment subject matter expertise in effectively defining requirements for three 
pending maritime capabilities. Moreover, this research is the first to evaluate the 


requirements needed for an effective system of complimentary non-lethal capabilities. 


Validates Tactics - Current Navy Tactics, Techniques and Procedures for anti- 
terrorism/force protection include only current capabilities. This research analyzes the 
integration of imminent capabilities into current tactics prior to acquisition, thereby 
providing warfighters with updated tactics as soon as capabilities are released. Included 
is an analysis of tactical effectiveness given varying rules of engagements, covering any 


geo-political situation. 
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Revolutionizes Vulnerability Assessments - Navy doctrine requires examination 





of likely enemy courses of action and friendly counteraction when conducting anti- 
terrorist/force protection planning. Current tactical-level wargaming efforts include table 
top discussions, which generally cover only a few possible courses of action. This thesis 
leverages high power computing to evaluate virtually any possible interaction with 
multiple threat-types, thereby generating a robust port vulnerability assessment. The 
methodology and model used in this research are quickly altered to support any port or 


choke point geography. 


In summary, this research examines newly fielded or not-yet-fielded technologies, 
meaning little performance data exists. Secondly, this research tackles complex 
interactions, including critical human factors, in emulating an intricate, real-world 
scenario. Lastly, this research attempts to capture and make sense of the multitude of 
possible outcomes. George E. P. Box said, “All models are wrong, but some are useful” 
(Box, 1979). Despite the initial limitations and complexities, this thesis produces 
valuable insights through application of proven operations research tools and techniques. 
However, this is nothing but a beginning. Through continued application of this work 
across the spectrum of non-lethal applications in maritime operations, we considerably 
enhance our efforts to preserve the freedom of the seas and ultimately, to bring our sailors 


safely home. 
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APPENDIX A. PYTHAGORAS COLOR COMPARISON TOOL 


PYTHAGORAS uses red-green-blue (RGB) color spectrum to differentiate 
‘sidedness’ or affiliations in a given scenario (see the PYTHAGORAS manual for further 
detail). This approach provides nearly unlimited flexibility in assigning multiple-sided 
forces and their relationships to one another. Because a user can quickly lose track of 
who is affiliated with whom, the PYTHAGORAS software package includes a Microsoft 
Excel document that allows the user to create the affiliations as he or she would in the 
PYTHAGORAS GUI, and then quickly determine the resultant relationships based on 
these inputs. 


Unit Friendly Enemy 


() Use 
Agent Name Rad | Red 


lueAHD | Oo | 255] 3 | 0 | 


3 |_ Oo | 
TargetHostile | 6 | 255] 0 [| o | o | of of of ofofofpeot of] of 4 | 





Figure 28. Sidedness RGB Inputs for PYTHAGORAS Scenario 


Figure 28 displays the sidedness RGB levels inputted into the PYTHAGORAS 
scenario for this research. The first four agents listed represent the three non-lethal 
capabilities and the lethal capability of the blue ship. The remaining three listed agents 
do not represent an actual target, but an identification state of an inbound target. 
Independent of its ground truth identification, the blue ship will always hold targets in 


one of these three states. 


[BiueAHD | BlueOD | BlueWM | BlueLethal | TargetNeutral | TargetSuspect | TargetHostile | 
BiueAHD |__| UN | UNT | UN | ENEMY | ENEMY | ___ ENEMY _| 
BlueoD {UN | — | UNIT | UNIT | ENEMY | ENEMY | ___ENEMY _| 
Bluewm | __UNIT_|_UNT | | UNIT | _ENEMY_| ENEMY | __ ENEMY _| 


BlueLethal | UNIT | UNIT | UNIT [OT ENEMY [ENEMY | ENEMY 
TargetNeutral | ENEMY | ENEMY] fT 
TargetSuspect_| ENEMY | ENEMY| ENEMY |] | 
targetHostile | ENEMY | ENEMY| ENEMY | ENEMY [Of 





Figure 29. Agent Affiliations for PYTHAGORAS Scenario 


Given the inputs, the PYTHAGORAS color comparison tool automatically 


generates the associations of the listed agents to one another. Figure 29 displays the 
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affiliations of this thesis. An agent can ‘see’ another agent in one of four ways: Unit, 
friend, enemy or neutral. In this example, the AHD and OD see all targets as enemy, 
meaning all targets are eligible to be engaged with AHD and OD non-lethal capabilities. 
WMs will only be employed against targets identified as suspect or hostile, while lethal 


force will only be used against hostile targets. 


The targets view all blue assets as ‘enemy.’ This does not mean that every target 
type attacks the blue ship. Several subsequent decision rules invoked when an agent 
interacts with another agent are based on the affiliation between the two. Therefore, the 
categorization of ‘enemy’ is used to define subsequent sensing and movement triggers. 
For example, when a neutral target is deterred, its movement desire is changed to move 
away from the nearest ‘enemy,’ which is defined in Figure 29 as the blue ship. The blank 
spaces relate to an affiliation of neutral. The neutral association of all targets to one 
another is based on the initial assumption that all agent classes move, decide and engage 


independently from one another. 
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APPENDIX B. RUBY SCRIPT FOR DATA FARMING 


As discussed in Chapter III, Ruby scripting code is used to implement the NOLH 


DOE in a distributed simulation. The script serves to accomplish 5 main tasks: 
1) Take the appropriate design point from the NOLH.xls spreadsheet. 


2) Update the baseline PYTHAGORAS XML scenario file with the current 
design point values, taken from the NOLH.xls spreadsheet. 


3) Update a text input file to be used to execute the Pythagoras batch engine. The 
input file includes all information needed to run the Pyth.Eng.bat including seed number, 


index number, and output storage location. 


4) Execute the design point in PYTHAGORAS for the desired number of 
iterations (designated in the input file) using the Pyth.Eng.bat file included in the 
PYTHAGORAS files. 


5) Repeat the process for the next design point. 


The specific code written to accomplish this is provided below: 


Line Code 
1 File Name: FarmPyth.rb 
2 #Add-in's required to manipulate XML files with Ruby 
3 require "rexml/document" 
4 include REXML 
5 #Method to update the XML scenario file with a design point. The input parameters for the 
6 
7 
8 





current design point are passed to the method in the valueArray. Each factor is represented by 
an element in the array. The xpath in each [ ] set represents the xpath to the specific factor in 
the Pythagoras.xml scenario file.# 

9 def updateScenarioFile(valueArray) 


10 #loads the sceanio and converts it to REXML for manipulation 

11 baseScenarioFile = File.new(‘E:\Norfolk.xml’, "r") 

12 baseScenarioDoc = Document.new(baseScenarioFile) 

13 baseScenarioFile.close 

14 

15 #updates AHDRange 

16 baseScenarioDoc.elements["pythagoras/weaponList/weapon/maxRange" ].text = 


17 valueArray[0] 

18 #updates OD Range 

19 baseScenarioDoc.elements["pythagoras/weaponList/weapon[2]/maxRange"].text = 
20 valueArray[1] 
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Line Code 
21 File Name: FarmPyth.rb 
22 #Add-in's required to manipulate XML files with Ruby 
23 require "rexml/document" 
24 include REXML 
25 #Method to update the XML scenario file with a design point. The input parameters for the 
26 current design point are passed to the method in the valueArray. Each factor is represented by 
27 an element in the array. The xpath in each [ ] set represents the xpath to the specific factor in 
28 the Pythagoras.xml scenario file.# 
29 def updateScenarioFile(valueArray) 
30 #loads the sceanio and converts it to REXML for manipulation 
31 baseScenarioFile = File.new('E:\Norfolk.xml’, "r") 
32 baseScenarioDoc = Document.new(baseScenarioFile) 
33 baseScenarioFile.close 
34 
35 #updates AHDRange 
36 baseScenarioDoc.elements["pythagoras/weaponList/weapon/maxRange" ].text = 
37 valueArray[0] 
38 #updates OD Range 
39 baseScenarioDoc.elements["pythagoras/weaponList/weapon[2]|/maxRange"].text = 
40 valueArray[1] 
41 #updates WM Range 
42 baseScenarioDoc.elements["pythagoras/weaponList/weapon[S5 ]/maxRange"].text = 
43 valueArray[2] 
44 #updates AHD Firing Rate 
45 baseScenarioDoc.elements["pythagoras/weaponList/weapon/shotsPerTimeStep" ].text = 
46 valueArray[3] 
47 #updates OD Red Radius 
48 baseScenarioDoc.elements["pythagoras/sidednessList/sidedness[2]/enemy/radius"].text = 
49 valueArray[4] 
50 #updates WM Red Radius 
51 baseScenarioDoc.elements["pythagoras/sidednessList/sidedness[5 ]/enemy/radius"].text = 
52 valueArray[5] 
53 #updates Lethal Red Radius 
54 baseScenarioDoc.elements["pythagoras/sidednessList/sidedness[3 ]/enemy/radius"].text = 
55 valueArray[6] 
56 #updates TargetHostile Color Vulnerability 
57 baseScenarioDoc.elements["pythagoras/agentList/agent[6]/colorVulnerability/colorVulnerabilit 
58 yValue"].text = valueArray[7] 
59 #updates TargetHostile[2] Color Vulnerability 
60 baseScenarioDoc.elements["pythagoras/agentList/agent[7]/colorVulnerability/colorVulnerabilit 
61 yValue"].text = valueArray[8] 
62 #updates TargetNeutral Color Vulnerability 
63 baseScenarioDoc.elements["pythagoras/agentList/agent[8 ]/colorVulnerability/colorVulnerabilit 
64 yValue"].text = valueArray[9] 
65 #updates TargetNeutral[2] Color Vulnerability 
66 baseScenarioDoc.elements["pythagoras/agentList/agent[9 |/colorVulnerability/colorVulnerabilit 
67 yValue"].text = valueArray[10] 
68 #updates TargetNeutralFisher Color Vulnerability 
69 baseScenarioDoc.elements["pythagoras/agentList/agent[ 10]/colorVulnerability/colorVulnerabil 
70 ityValue"].text = valueArray[11] 
71 #updates TargetSuspectFisher Color Vulnerability 
72 baseScenarioDoc.elements["pythagoras/agentList/agent{[1 1 ]/colorVulnerability/colorVulnerabil 
73 ity Value" ].text = valueArray[12] 
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Line Code 
74 #updates TargetHostile Attribute Vulnerability 
75 baseScenarioDoc.elements["pythagoras/agentList/agent[6]/attribute Vulnerability/attribute Vuln 
76 erability Value"].text = valueArray[13] 
77 #updates TargetHostile[2] Attribute Vulnerability 
78 baseScenarioDoc.elements["pythagoras/agentList/agent[7 ]/attribute Vulnerability/attribute Vuln 
719 erability Value"].text = valueArray[14] 
80 #updates TargetNeutral Attribute Vulnerability 
81 baseScenarioDoc.elements["pythagoras/agentList/agent[8 ]/attribute Vulnerability/attribute Vuln 
82 erability Value"].text = valueArray[15] 
83 #updates TargetNeutral[2] Attribute Vulnerability 
84 baseScenarioDoc.elements["pythagoras/agentList/agent[9 ]/attribute Vulnerability/attribute Vuln 
85 erability Value"].text = valueArray[16] 
86 #updates TargetNeutralFisher Attribute Vulnerability 
87 baseScenarioDoc.elements["pythagoras/agentList/agent[10]/attribute Vulnerability/attributeVul 
88 nerability Value" ].text = valueArray[17] 
89 #updates TargetSuspectFisher Attribute Vulnerability 
90 baseScenarioDoc.elements["pythagoras/agentList/agent[ 1 1 ]/attribute Vulnerability/attributeVul 
91 nerability Value" ].text = valueArray[18] 
92 #updates TargetHostile Instance 
93 baseScenarioDoc.elements["pythagoras/agentList/agent[6]/instance"].text = valueArray[19] 
94 #updates TargetHostile[2] Instance 
95 baseScenarioDoc.elements["pythagoras/agentList/agent[7]/instance"].text = valueArray[20] 
96 #updates TargetNeutralFisher Instance 
97 baseScenarioDoc.elements["pythagoras/agentList/agent[10]/instance"].text = valueArray[21] 
98 #updates TargetSuspectFisher Instance 
99 baseScenarioDoc.elements["pythagoras/agentList/agent[11]/instance"].text = valueArray[22] 
100 #updates TargetHostile Speed 
101 | baseScenarioDoc.elements["pythagoras/agentList/agent[6]/speed/avgSpeed"].text = 
102 | valueArray[23] 
103 #updates TargetHostile[2] Speed 
104 | baseScenarioDoc.elements["pythagoras/agentList/agent[7]/speed/avgSpeed"].text = 
105 | valueArray[24] 
106 #updates TargetNeutral Speed 
107 | baseScenarioDoc.elements["pythagoras/agentList/agent[8 |/speed/avgSpeed"].text = 
108 valueArray[25] 
109 #updates TargetNeutral[2] Speed 
110 | baseScenarioDoc.elements["pythagoras/agentList/agent[9 |/speed/avgSpeed"].text = 
111 valueArray[26] 
112 #updates TargetHostile Alpha Less Than Trigger Value 
113 | BaseScenarioDoc.elements["pythagoras/agentList/agent[6]/triggerList/alphaLossTrigger/lessT 
114 | han").text = valueArray[27] 
115 #updates TargetHostile[2] Alpha Less Than Trigger Value 
116 | baseScenarioDoc.elements["pythagoras/agentList/agent[7 ]/triggerList/alphaLossTrigger/lessT 
117 | han").text = valueArray[28] 
118 #updates TargetNeutral Alpha Less Than Trigger Value 
119 | baseScenarioDoc.elements["pythagoras/agentList/agent[8 ]/triggerList/alphaLossTrigger/lessT 
120 | han").text = valueArray[29] 
121 #updates TargetNeutral[2] Alpha Less Than Trigger Value 
122 | baseScenarioDoc.elements["pythagoras/agentList/agent[9 |/triggerList/alphaLossTrigger/lessT 
123 han"].text = valueArray[30] 
124 #updates TargetNeutralFisher Alpha Less Than Trigger Value 
125 | baseScenarioDoc.elements["pythagoras/agentList/agent[ 10]/triggerList/alphaLossTrigger/less 
126 | Than").text = valueArray[31] 
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Code 











#updates TargetSuspectFisher Alpha Less Than Trigger Value 
baseScenarioDoc.elements["pythagoras/agentList/agent[ 1 1 ]/triggerList/alphaLossTrigger/less 
Than"].text = valueArray[32] 

#writes the changes to the .xml file for implementation 

file = File.new('E:\Norfolk.xml’, "w" 

baseScenarioDoc.write(file) 

file.close 
end 
#Opens the Design of Experiment file and strips the factor names from Ist row 
$inputFile = File.new('C:\Pythagoras\inputData.csv', "r") 
$columnHeaders = $inputFile.readline.strip.split(",") 

#Loop to iterate through the design points. 'X'.times represents the number of design #points 
to be run. 

100.times do IdesignPointl 

#Sets beginning seed and index for current design point 

$seed = (designPoint* 100) + 1 

$index = (designPoint* 100) + 1 

#Opens old pythenginput file to change seed and index. The pythinput file feeds the 
#PythEng.bat file the necessary inputs to run. 

pythenginput = File.open(‘C:\Pythagoras\pythenginput.txt’, "r+") 

array = pythenginput.readlines 

pythenginput.close 

array[2] = "#{$seed}\n" 

array[3] = "#{$index }\n" 

#Writes new pythenginput file for current design point 

logstream = File.new('C:\Pythagoras\pythenginput.txt’, "w" 

7.times {lil logstream.write(array[i]) } 

logstream.close 

#Updates XML scenario file with current design point input parameters 

valueArray = $inputFile.readline.strip.split(",") 

updateScenarioFile(valueArray) 

puts "Attempting to calculate designPoint " + designPoint.to_s 

#executes Pythagoras (both pytheng.bat and pythenginput.txt must be in same folder as 
#FarmPyth.rb) 

puts “PythEng.bat < pythenginput.txt 


print "Just finished " + designPoint.to_s 
end 
print "Batch Complete\n" 
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APPENDIX C. VBA SCRIPT FOR DATA CLEANING 


As discussed in Chapter 3, a sizable amount of data cleaning was needed in order 
to produce a single file to be used for data analysis. The following sub-programs coded 
using VBA and Microsoft Excel macro recording accomplish the required data cleaning 


for this research. 


DPlterateQ): This class takes the 100 design points of the NOLH.xls spreadsheet 
and creates blank rows in between each design point which correspond to the number of 


desired iterations per design point. 





Sub DPlterate() 


"DPlterate Macro 
‘Macro recorded 8/23/2006 by Lisa R. Sickinger 
' Creates rows in between each design point for iterations 


' 


For i= 0 To 256 
Cells(((100 * i) + 2), 2).Select ''100' is number of iterations per run 
For j = 0 To 98 ''98' is number of iterations per run - 2 
Selection.EntireRow. Insert 
Next j 
Next i 








End Sub 





RowCopy(): This copies the input data from each design point to the newly 


created blank rows associated with that design point. 





Sub RowCopy() 


"RowCopy Macro 
"Macro recorded 8/23/2006 by Lisa R. Sickinger 
Copies each design point applicable number of times 


1 


For i= 0 To 256 
Range((Cells(((100 * i) + 2), 2)), (Cells(((100 * 1) + 2), 34))).Select 
''100' = number of design points 
''34' = number of factors + | 
Selection.AutoFill Destination:=Range((Cells(((100 * i) + 2), 2)), (Cells(((100 * 1) + 101), 34))), 
Type:=x1FillDefault 
''101' = number of design points + 1 
Range((Cells(((100 * i) + 2), 2)), (Cells(((100 * 1) + 101), 34))).Select 
Next i 
End Sub 
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Consolidate(): This class takes the 25,700 individual AGB files created from the 
time series MOEs and consolidates them to 257 files, one for each design point. An 
individual file has a header with agent information, followed by 100 rows of raw data, 
corresponding to the output data of the 100 iterations for that design point. Because 
different design points have different number of agents, the program is flexible to 
construct the correct number of columns for each design point file. 


Sub Consolidate() 


‘ Consolidate Macro 
‘Macro recorded 8/20/2006 by Lisa Sickinger 
' change name of file to reflect file that starts current design point 
begDP = Cells(3, 3) ‘ first dp 
endDP = Cells(4, 3) ‘ last dp 
For dp = begDP To endDP 
Name = (dp * 100) - 99 
Workbooks.Open Filename:= _ 
"C:\Documents and Settings\SEEDLab\NorfolkMOE\DP" & dp & "Output.x1s" 
Workbooks.OpenText Filename:= _ 
"C:\Documents and Settings\SEEDLab\NorfolkMOE\Norfolk_" & Name & _ "_Agent 
ABG.AgentAlpha,Beta,Gamma" _ 
, Origin:=437, StartRow:=1, DataType:=xIDelimited, TextQualifier:= _ 
xlDoubleQuote, ConsecutiveDelimiter:=False, Tab:=False, Semicolon:=False _ 
, Comma:=True, Space:=False, Other:=False, FieldInfo:=Array(Array(1, 1), _ 
Array(2, 1), Array(3, 1), Array(4, 1), Array(5, 1), Array(6, 1), Array(7, 1), Array(8, 1), _ 
Array(9, 1), Array(10, 1), Array(11, 1), Array(12, 1), Array(13, 1), Array(14, 1), Array(15 _ 
, 1), Array(16, 1), Array(17, 1), Array(18, 1), Array(19, 1), Array(20, 1), Array(21, 1), _ 
Array(22, 1), Array(23, 1), Array(24, 1), Array(25, 1), Array(26, 1), Array(27, 1), Array( _ 
28, 1), Array(29, 1), Array(30, 1), Array(31, 1), Array(32, 1), Array(33, 1), Array(34, 1), _ 
Array(35, 1), Array(36, 1), Array(37, 1), Array(38, 1), Array(39, 1), Array(40, 1), Array( _ 
41, 1), Array(42, 1), Array(43, 1), Array(44, 1), Array(45, 1), Array(46, 1), Array(47, 1), _ 
Array(48, 1), Array(49, 1), Array(50, 1), Array(51, 1), Array(52, 1), Array(53, 1), Array( _ 
54, 1), Array(55, 1), Array(56, 1), Array(57, 1), Array(58, 1), Array(59, 1), Array(60, 1), _ 
Array(61, 1), Array(62, 1), Array(63, 1), Array(64, 1), Array(65, 1), Array(66, 1), Array( _ 
67, 1), Array(68, 1), Array(69, 1)), TrailingMinusNumbers:=True 
Range("B2:CC4").Select 
Selection.Copy 
‘change name of active window 
Windows("DP" & dp & "Output").Activate 
Selection.PasteSpecial Paste:=x1Paste Values, Operation:=xlNone, SkipBlanks _ 
:=False, Transpose:=False 
change name of active window 
Windows("Norfolk_" & Name & "_Agent ABG.AgentAlpha, Beta,Gamma").Activate 
ActiveWindow.Close 
‘change i = x01 To (x+1)00 
For i = ((dp * 100) - 98) To (dp * 100) 
Workbooks.OpenText Filename:= _ 
"C:\Documents and Settings\SEEDLab\NorfolkMOE\Norfolk_" & i & "_Agent 
ABG.AgentAlpha,Beta,Gamma" _ 
, Origin:=437, StartRow:=1, DataType:=xIDelimited, TextQualifier:= _ 
xlDoubleQuote, ConsecutiveDelimiter:=False, Tab:=False, Semicolon:=False _ 
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, Comma:=True, Space:=False, Other:=False, FieldInfo:=Array(Array(1, 1), _ 
Array(2, 1), Array(3, 1), Array(4, 1), Array(5, 1), Array(6, 1), Array(7, 1), Array(8, 1), _ 
Array(9, 1), Array(10, 1), Array(11, 1), Array(12, 1), Array(13, 1), Array(14, 1), Array(15 _ 
, 1), Array(16, 1), Array(17, 1), Array(18, 1), Array(19, 1), Array(20, 1), Array(21, 1), _ 
Array(22, 1), Array(23, 1), Array(24, 1), Array(25, 1), Array(26, 1), Array(27, 1), Array( _ 
28, 1), Array(29, 1), Array(30, 1), Array(31, 1), Array(32, 1), Array(33, 1), Array(34, 1), _ 
Array(35, 1), Array(36, 1), Array(37, 1), Array(38, 1), Array(39, 1), Array(40, 1), Array( _ 
41, 1), Array(42, 1), Array(43, 1), Array(44, 1), Array(45, 1), Array(46, 1), Array(47, 1), _ 
Array(48, 1), Array(49, 1), Array(50, 1), Array(51, 1), Array(52, 1), Array(53, 1), Array( _ 
54, 1), Array(55, 1), Array(56, 1), Array(57, 1), Array(58, 1), Array(59, 1), Array(60, 1), _ 
Array(61, 1), Array(62, 1), Array(63, 1), Array(64, 1), Array(65, 1), Array(66, 1), Array( _ 
67, 1), Array(68, 1)), TrailingMinusNumbers:=True 

Range("B4:CC4").Select 

Selection.Copy 

‘change name of active window 

Windows("DP" & dp & "Output.xls").Activate 

' change (i-x) to design point start -1 

Cells((i - ((dp * 100) - 100) + 2), 1).Select 

Selection.PasteSpecial Paste:=x1Paste Values, Operation:=xlNone, SkipBlanks _ 
:=False, Transpose:=False 

Windows("Norfolk_" & i & "_Agent ABG.AgentA]Ipha,Beta,Gamma").Activate 

Active Window.Close 

Next i 

Windows("DP" & dp & "Output").Activate 

ActiveWorkbook.save 

Active Window.Close 

Next dp 

End Sub 











Truncate(): When it is time to create the formulas needed to generate the MOEs 
of interest from the raw data, a problem arises when using conditional statements for an 
agent class type. Each agent has its class name listed followed by a number indicating 
the specific agent number for that run. An example is Hostile2Target (5), where the data 
represents the 5"" agent for that run, which is of class type Hostile2. In order to perform 
conditional calculations for all agents of a class type, the specific agent identification 
number needs to be truncated. The following program truncates the agent identification 
from the header line of the design point file. This is done by defining the header column 
as delimited data where the delimiter is the first open parenthesis. Therefore, the newly 
formed header will include all previous information, regardless of string length, minus all 


text following the first parenthesis. This process is repeated for all 257 design point files. 
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Sub Truncate() 


"Truncate Macro 
"Macro recorded 8/27/2006 by SEEDLab 


' 


For dp = 1 To 257 
Workbooks.Open Filename:= _ 
"C:\Documents and Settings\SEEDLab\NorfolkMOE\DPOutput\DP" & dp & "Output.xls" 


Rows("1:1").Select 

Selection.Copy 

Sheets("Sheet2").Select 

Selection.PasteSpecial Paste:=x]PasteAll, Operation:=xlNone, SkipBlanks:= _ 
False, Transpose:=True 

Application.CutCopyMode = False 

Selection.TextToColumns Destination:=Range("A1"), DataType:=x1Delimited, _ 
TextQualifier:=x]DoubleQuote, ConsecutiveDelimiter:=False, Tab:=False, _ 
Semicolon:=False, Comma:=False, Space:=False, Other:=True, OtherChar _ 
:="(", FieldInfo:=Array(Array(1, 1), Array(2, 1)), TrailingMinusNumbers:=True 

Selection.Copy 

Sheets("Sheet1").Select 

Range("A1").Select 

Selection.PasteSpecial Paste:=x]PasteAll, Operation:=xlNone, SkipBlanks:= _ 
False, Transpose:=True 

ActiveWorkbook.Save 

ActiveWorkbook.Close 

Next dp 








End Sub 


InsertColQ: First, a clone of the first design point file is created and it is renamed 
“MasterOutput.” This file is used to create all the MOE formulas/columns of interest. 
Once the desired MOE columns are created, the following code inserts new columns in 
each design point file that correspond to the number of MOE formula columns. These 
columns are added the beginning of each design point file, so that when they are later 
merged, the same columns will be copied from each design point file, regardless of how 


many columns each file contains. 





Sub Insertcol() 


‘Insertcol Macro 
"Macro recorded 8/27/2006 by SEEDLab 


' 


For dp = 1 To 257 
Workbooks.Open Filename:= _ 
"C:\Documents and Settings\SEEDLab\NorfolkMOE\DPOutput\DP" & dp & "Output.xls" 
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For i= 1 To 33 
Columns("A:A").Select 
Selection. Insert Shift:=xIToRight 

Next i 

ActiveWorkbook.Save 

ActiveWorkbook.Close 

Next dp 
End Sub 








Formulas(Q): This program copies the formulas created in the MOE columns of 
the MasterOutput file and places them in the newly formed MOE columns of each design 


point file. The result is automatically calculated MOEs for each design point file. 





Sub Formulas() 


"Formulas Macro 
"Macro recorded 8/27/2006 by SEEDLab 


For dp = 1 To 257 


Range("A1:AG102").Select 

Selection.Copy 

Workbooks.Open Filename:= _ 
"C:\Documents and Settings\SEEDLab\NorfolkMOE\DPOutput\DP" & dp & "Output.x1s" 

Range("A1").Select 

Selection.PasteSpecial Paste:=xlPasteFormulas, Operation:=xlNone, _ 
SkipBlanks:=False, Transpose:=False 

Application.CutCopyMode = False 

Calculate 

ActiveWorkbook.Save 

ActiveWorkbook.Close 

Windows("OutputMaster"). Activate 

Next dp 
End Sub 











Merge(): This program copies all MOE columns from each design point file and 
pastes them alongside the corresponding input data created using the DPlIterate and 
RowCopy programs. The result is a single file with 25,700 rows of data. The first 33 
columns represent the input parameters while the remaining columns represent the MOE 
data, or output values. The file is now ready for analysis using one’s favorite statistical 


software package. 
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Sub merge() 
For dp = 1 To 257 


Workbooks.Open Filename:= _ 
"C:\Documents and Settings\SEEDLab\NorfolkMOE\DPOutput\DP" & dp & "Output.x1s" 
Range("B3:AG102").Select 
Selection.Copy 
Windows("SingleOutput").activate 
Cells(((dp * 100) - 100) + 2, 35).Select 
Selection.PasteSpecial Paste:=xlPaste Values, Operation:=xlNone, SkipBlanks _ 
:=False, Transpose:=False 
Windows("DP" & dp & "Output").activate 
ActiveWorkbook.Save 
ActiveWorkbook.Close 
Windows("SingleOutput").activate 


Next dp 


End Sub 
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APPENDIX D. MILITARY OPERATIONS RESEARCH SOCIETY 
(MORS) TISDALE COMPETITION PRESENTATION 


The following slides were presented to a committee of faculty at the Naval 
Postgraduate School for the Military Operations Research Society (MORS) Stephen A. 
Tisdale Graduate Research Award. This award recognizes high-quality research of 
immediate or near-term value to the defense of the United States and its allies. This 
author was one of five finalists selected for the competition based on the potential impact 


of this thesis work on the Department of Defense. 
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Research Questions 
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Supporting questions: 


Is Multi-Agent Simulation (MAS) an appropriate tool? 
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What are the geographical effects? 


How to identify threats from non-threats? 
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¢ Uses Multiple Analysis Techniques 
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— Multiple Regression 
— Classification Trees 


— Data Visualization 
* Contour Plots 
¢ 3-D Surface Plots 
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